A Small Example

This example covers the generation of test data and oracles and the combination of features using states.

Let's suppose that we have to create two simple features for a web application: "Login" and "Add Product". After discussing them with stakeholders and the team, we created the following Concordia specification (see the two tabs):

Feature: Login
  As a user
  I would like to authenticate myself
  In order to access the application

Scenario: Successful login
  Given that I can see the login screen
  When I enter with valid credentials
  Then I can access the application's main screen
  
  Variant: Login with username and password
    Given that I visit the [Login Screen]
    When I fill {Username}
      And I fill {Password}
      And I click on {OK}
    Then I see "Welcome"
      And I have a ~user logged in~

Table: Users
  | username | password  |
  | bob      | 123456    |
  | alice    | 4l1c3pass |

UI Element: Username
  - required
    Otherwise I must see "Please inform the username."
  - value comes from "SELECT username FROM [Users]"
    Otherwise I must see "Invalid username."    

UI Element: Password
  - required
    Otherwise I must see "Please inform the password."
  - value comes from "SELECT password FROM [Users] WHERE username = {Username}"
    Otherwise I must see "Invalid password."

UI Element: OK
  - type is button
  
Constants:
  - "Login Screen" is "/login"

import "login.feature"

Feature: Add Product
  As an employee
  I would like to add a product
  In order to manage its data and sale it
  
Scenario: Add a new product with basic data
  Given that I am logged in
  When I add a product with SKU, description, department, price, and quantity
  Then I am able to register it
  
  Variant: Basic data
    Given that I have a ~user logged in~
      and I visit the [Product Screen]
    When I fill {SKU}
      and I fill {Description}
      and I fill {Price}
      and I fill {Quantity}
      and I click on {Save}
    Then I have ~product registered~
      and I see "Saved."

UI Element: SKU
 - required
   Otherwise I see "Please inform the SKU."
 - format is "/[A-Z]{3}\-[0-9]{3}/"
   Otherwise I see "Invalid SKU format. Please use 3 letters, dash, 3 numbers." 
    
UI Element: Description
  - minimum length is 2
    Otherwise I see "Description needs at least 2 characters."
  - maximum length is 100
    Otherwise I see "Description must have at most 100 characters."    
    
UI Element: Price
  - required 
    Otherwise I see "Please inform the price."  
  - minimum value is 0.01
    Otherwise I see "Minimum price is one cent."
    
UI Element: Quantity
  - data type is integer

UI Element: Save
  - type is button    

Constants:
  "Product Screen" is "/products/new"

These features could correspond to the following sketches:

There is an experimental application that can generate User Interface Prototypes (UIP) from a Concordia specification. The current version has a plug-in to create HTML-based UIPs and it's really easy to adapt it for other technologies.

The specification helps stakeholders, analysis, testers and developers to create a shared understanding about the application and its business rules. It helps with the requirements validation, knowledge documentation, project design, implementation and testing.

Different from Use Cases and other requirements specification formats, with Concordia you probably don't need to define different scenarios for validation. Instead of having to manually define a set scenarios for validation, you just need to define how the UI elements should behavior - in their declaration - and the compiler will generate different scenarios for you in the form of Test Cases. Whether your application has a lot of such rules, your team will certainly benefit from it.

<TO-DO (UNDER CONSTRUCTION)>

Remember that a Variant express a possible interaction between a user and the application in order to complete its (business-focused) Scenario. lt follows the GWT syntax and always uses the word "I" to represent the user or user role denoted in the Feature's description.

Let's run Concordia Compiler with a seed to produce the same results over and over again:

npx concordia --seed="example" --no-run --no-result

It will generate login.testcase with the following content:

<TO-DO (UNDER CONSTRUCTION)>

If a seed is not given, Concordia Compiler assumes the current date and time. The seed is always printed in the console so that you or your team can reproduce the same paths, test data, and test oracle that were able to expose a bug in your application. Here we are giving it for a sake of reproduction.

Using the same seed over and over again will make Concordia Compiler produce the same results. Options no-run and no-result avoid running the test scripts and getting execution results.

Execution without generation

This will not generate test cases or test scripts, but it will execute them and get their results:

$ concordia --plugin=codeceptjs --no-test-case --no-script

Output

login.testcase:

# Generated with ❤ by Concordia
#
# THIS IS A GENERATED FILE - MODIFICATIONS CAN BE LOST !

import "login-en.feature"

@generated
@scenario(1)
@variant(1)
Test case: Successful login with valid credentials - 1
  Given that I am in the "http://localhost/login"  # [Login Screen]
  When i fill <username> with "*RM)O,"  # invalid: inexistent element
    And i fill <password> with ""  # valid: last element
    And I click on <ok>  # {OK}
  Then I must see "Invalid username"  # from <username>

@generated
@scenario(1)
@variant(1)
Test case: Successful login with valid credentials - 2
  Given that I am in the "http://localhost/login"  # [Login Screen]
  When i fill <username> with "bob"  # valid: filled
    And i fill <password> with -8655972838932479  # invalid: inexistent element
    And I click on <ok>  # {OK}
  Then I must see "Invalid password"  # from <password>

@generated
@scenario(1)
@variant(1)
Test case: Successful login with valid credentials - 3
  Given that I am in the "http://localhost/login"  # [Login Screen]
  When i fill <username> with "alice"  # valid: random element
    And i fill <password> with "4l1c3pass"  # valid: random element
    And I click on <ok>  # {OK}
  Then I see "Welcome"

@generated
@scenario(1)
@variant(1)
Test case: Successful login with valid credentials - 4
  Given that I am in the "http://localhost/login"  # [Login Screen]
  When i fill <username> with "alice"  # valid: last element
    And i fill <password> with "4l1c3pass"  # valid: filled
    And I click on <ok>  # {OK}
  Then I see "Welcome"

@generated
@fail
@scenario(1)
@variant(1)
Test case: Successful login with valid credentials - 5
  Given that I am in the "http://localhost/login"  # [Login Screen]
  When i fill <username> with ""  # invalid: not filled
    And i fill <password> with ""  # invalid: not filled
    And I click on <ok>  # {OK}
  Then I see "Welcome"

@generated
@scenario(1)
@variant(1)
Test case: Successful login with valid credentials - 6
  Given that I am in the "http://localhost/login"  # [Login Screen]
  When i fill <username> with "bob"  # valid: first element
    And i fill <password> with 123456  # valid: first element
    And I click on <ok>  # {OK}
  Then I see "Welcome"

test/login.js:

// Generated with ❤ by Concordia
// source: c:\code\tmp\concordia-test-pt\login-en.testcase
//
// THIS IS A GENERATED FILE - MODIFICATIONS CAN BE LOST !

Feature("Login");

Scenario("Successful login | Successful login with valid credentials - 1", (I) => {
    I.amOnPage("http://localhost/login"); // (61,5)  [Login Screen]
    I.fillField("username", "*RMO,"); // (11,5)  invalid: inexistent element
    I.fillField("password", ""); // (12,7)  valid: last element
    I.click("ok"); // (64,7)  {OK}
    I.see("Invalid username"); // (14,5)  from <username>
});

Scenario("Successful login | Successful login with valid credentials - 2", (I) => {
    I.amOnPage("http://localhost/login"); // (61,5)  [Login Screen]
    I.fillField("username", "bob"); // (21,5)  valid: filled
    I.fillField("password", "-8655972838932479"); // (22,7)  invalid: inexistent element
    I.click("ok"); // (64,7)  {OK}
    I.see("Invalid password"); // (24,5)  from <password>
});

Scenario("Successful login | Successful login with valid credentials - 3", (I) => {
    I.amOnPage("http://localhost/login"); // (61,5)  [Login Screen]
    I.fillField("username", "alice"); // (31,5)  valid: random element
    I.fillField("password", "4l1c3pass"); // (32,7)  valid: random element
    I.click("ok"); // (64,7)  {OK}
    I.see("Welcome"); // (65,5)
});

Scenario("Successful login | Successful login with valid credentials - 4", (I) => {
    I.amOnPage("http://localhost/login"); // (61,5)  [Login Screen]
    I.fillField("username", "alice"); // (41,5)  valid: last element
    I.fillField("password", "4l1c3pass"); // (42,7)  valid: filled
    I.click("ok"); // (64,7)  {OK}
    I.see("Welcome"); // (65,5)
});

Scenario("Successful login | Successful login with valid credentials - 5", (I) => {
    I.amOnPage("http://localhost/login"); // (61,5)  [Login Screen]
    I.fillField("username", ""); // (52,5)  invalid: not filled
    I.fillField("password", ""); // (53,7)  invalid: not filled
    I.click("ok"); // (64,7)  {OK}
    I.see("Welcome"); // (65,5)
});

Scenario("Successful login | Successful login with valid credentials - 6", (I) => {
    I.amOnPage("http://localhost/login"); // (61,5)  [Login Screen]
    I.fillField("username", "bob"); // (62,5)  valid: first element
    I.fillField("password", "123456"); // (63,7)  valid: first element
    I.click("ok"); // (64,7)  {OK}
    I.see("Welcome"); // (65,5)
});

3. Analyze the results

Whether you ran the test scripts above, you probably saw they fail. That's because they didn't find a web application running at http://localhost/login or because the application was found but it did not match the expected behavior. You may adapt your application and run the tests again.

Concordia shows a report that indicates the failures' locations. They help you to decide if a failure was caused by the application under test (e.g., it did not behave as expected) or because of the requirements specification (e.g., it is outdated in relation to the application).

Now keeping your specification updated has a new clear benefit: you can use it to generate tests and discover existing defects in your application!

Execution without generation

Output

3. Analyze the results

See also