This example covers the generation of test data and oracles and the combination of features using states.
Let's suppose that we have to create two simple features for a web application: "Login" and "Add Product". After discussing them with stakeholders and the team, we created the following Concordia specification (see the two tabs):
Feature:LoginAsauserIwouldliketoauthenticatemyselfInordertoaccesstheapplicationScenario:SuccessfulloginGiventhatIcanseetheloginscreenWhenIenterwithvalidcredentialsThenIcanaccesstheapplication's main screen Variant: Login with username and password Given that I visit the [Login Screen] When I fill {Username} And I fill {Password} And I click on {OK} Then I see "Welcome" And I have a ~user logged in~Table: Users | username | password | | bob | 123456 | | alice | 4l1c3pass |UI Element: Username - required Otherwise I must see "Please inform the username." - value comes from "SELECT username FROM [Users]" Otherwise I must see "Invalid username." UI Element: Password - required Otherwise I must see "Please inform the password." - value comes from "SELECT password FROM [Users] WHERE username = {Username}" Otherwise I must see "Invalid password."UI Element: OK - type is buttonConstants: - "Login Screen" is "/login"
import"login.feature"Feature:AddProductAsanemployeeIwouldliketoaddaproductInordertomanageitsdataandsaleitScenario:AddanewproductwithbasicdataGiventhatIamloggedinWhenIaddaproductwithSKU,description,department,price,andquantityThenIamabletoregisteritVariant:BasicdataGiventhatIhavea~userloggedin~andIvisitthe [Product Screen]WhenIfill{SKU}andIfill{Description}andIfill{Price}andIfill{Quantity}andIclickon{Save}ThenIhave~productregistered~andIsee"Saved."UIElement:SKU-requiredOtherwiseIsee"Please inform the SKU."-formatis"/[A-Z]{3}\-[0-9]{3}/"OtherwiseIsee"Invalid SKU format. Please use 3 letters, dash, 3 numbers."UIElement:Description-minimumlengthis2OtherwiseIsee"Description needs at least 2 characters."-maximumlengthis100OtherwiseIsee"Description must have at most 100 characters."UIElement:Price-requiredOtherwiseIsee"Please inform the price."-minimumvalueis0.01OtherwiseIsee"Minimum price is one cent."UIElement:Quantity-datatypeisintegerUIElement:Save-typeisbuttonConstants:"Product Screen"is"/products/new"
These features could correspond to the following sketches:
There is an experimental application that can generate User Interface Prototypes (UIP) from a Concordia specification. The current version has a plug-in to create HTML-based UIPs and it's really easy to adapt it for other technologies.
The specification helps stakeholders, analysis, testers and developers to create a shared understanding about the application and its business rules. It helps with the requirements validation, knowledge documentation, project design, implementation and testing.
Different from Use Cases and other requirements specification formats, with Concordia you probably don't need to define different scenarios for validation. Instead of having to manually define a set scenarios for validation, you just need to define how the UI elements should behavior - in their declaration - and the compiler will generate different scenarios for you in the form of Test Cases. Whether your application has a lot of such rules, your team will certainly benefit from it.
<TO-DO (UNDER CONSTRUCTION)>
Remember that a Variant express a possible interaction between a user and the applicationin order to complete its (business-focused) Scenario. lt follows the GWT syntax and always uses the word "I" to represent the user or user role denoted in the Feature's description.
Let's run Concordia Compiler with a seed to produce the same results over and over again:
It will generate login.testcase with the following content:
<TO-DO (UNDER CONSTRUCTION)>
<TO-DO (UNDER CONSTRUCTION)>
If a seed is not given, Concordia Compiler assumes the current date and time. The seed is always printed in the console so that you or your team can reproduce the same paths, test data, and test oracle that were able to expose a bug in your application. Here we are giving it for a sake of reproduction.
Using the same seed over and over again will make Concordia Compiler produce the same results. Options no-run and no-result avoid running the test scripts and getting execution results.
Execution without generation
This will not generate test cases or test scripts, but it will execute them and get their results:
# Generated with ❤ by Concordia
#
# THIS IS A GENERATED FILE - MODIFICATIONS CAN BE LOST !
import "login-en.feature"
@generated
@scenario(1)
@variant(1)
Test case: Successful login with valid credentials - 1
Given that I am in the "http://localhost/login" # [Login Screen]
When i fill <username> with "*RM)O," # invalid: inexistent element
And i fill <password> with "" # valid: last element
And I click on <ok> # {OK}
Then I must see "Invalid username" # from <username>
@generated
@scenario(1)
@variant(1)
Test case: Successful login with valid credentials - 2
Given that I am in the "http://localhost/login" # [Login Screen]
When i fill <username> with "bob" # valid: filled
And i fill <password> with -8655972838932479 # invalid: inexistent element
And I click on <ok> # {OK}
Then I must see "Invalid password" # from <password>
@generated
@scenario(1)
@variant(1)
Test case: Successful login with valid credentials - 3
Given that I am in the "http://localhost/login" # [Login Screen]
When i fill <username> with "alice" # valid: random element
And i fill <password> with "4l1c3pass" # valid: random element
And I click on <ok> # {OK}
Then I see "Welcome"
@generated
@scenario(1)
@variant(1)
Test case: Successful login with valid credentials - 4
Given that I am in the "http://localhost/login" # [Login Screen]
When i fill <username> with "alice" # valid: last element
And i fill <password> with "4l1c3pass" # valid: filled
And I click on <ok> # {OK}
Then I see "Welcome"
@generated
@fail
@scenario(1)
@variant(1)
Test case: Successful login with valid credentials - 5
Given that I am in the "http://localhost/login" # [Login Screen]
When i fill <username> with "" # invalid: not filled
And i fill <password> with "" # invalid: not filled
And I click on <ok> # {OK}
Then I see "Welcome"
@generated
@scenario(1)
@variant(1)
Test case: Successful login with valid credentials - 6
Given that I am in the "http://localhost/login" # [Login Screen]
When i fill <username> with "bob" # valid: first element
And i fill <password> with 123456 # valid: first element
And I click on <ok> # {OK}
Then I see "Welcome"
test/login.js:
// Generated with ❤ by Concordia// source: c:\code\tmp\concordia-test-pt\login-en.testcase//// THIS IS A GENERATED FILE - MODIFICATIONS CAN BE LOST !Feature("Login");Scenario("Successful login | Successful login with valid credentials - 1", (I) => {I.amOnPage("http://localhost/login"); // (61,5) [Login Screen]I.fillField("username","*RMO,"); // (11,5) invalid: inexistent elementI.fillField("password",""); // (12,7) valid: last elementI.click("ok"); // (64,7) {OK}I.see("Invalid username"); // (14,5) from <username>});Scenario("Successful login | Successful login with valid credentials - 2", (I) => {I.amOnPage("http://localhost/login"); // (61,5) [Login Screen]I.fillField("username","bob"); // (21,5) valid: filledI.fillField("password","-8655972838932479"); // (22,7) invalid: inexistent elementI.click("ok"); // (64,7) {OK}I.see("Invalid password"); // (24,5) from <password>});Scenario("Successful login | Successful login with valid credentials - 3", (I) => {I.amOnPage("http://localhost/login"); // (61,5) [Login Screen]I.fillField("username","alice"); // (31,5) valid: random elementI.fillField("password","4l1c3pass"); // (32,7) valid: random elementI.click("ok"); // (64,7) {OK}I.see("Welcome"); // (65,5)});Scenario("Successful login | Successful login with valid credentials - 4", (I) => {I.amOnPage("http://localhost/login"); // (61,5) [Login Screen]I.fillField("username","alice"); // (41,5) valid: last elementI.fillField("password","4l1c3pass"); // (42,7) valid: filledI.click("ok"); // (64,7) {OK}I.see("Welcome"); // (65,5)});Scenario("Successful login | Successful login with valid credentials - 5", (I) => {I.amOnPage("http://localhost/login"); // (61,5) [Login Screen]I.fillField("username",""); // (52,5) invalid: not filledI.fillField("password",""); // (53,7) invalid: not filledI.click("ok"); // (64,7) {OK}I.see("Welcome"); // (65,5)});Scenario("Successful login | Successful login with valid credentials - 6", (I) => {I.amOnPage("http://localhost/login"); // (61,5) [Login Screen]I.fillField("username","bob"); // (62,5) valid: first elementI.fillField("password","123456"); // (63,7) valid: first elementI.click("ok"); // (64,7) {OK}I.see("Welcome"); // (65,5)});
3. Analyze the results
Whether you ran the test scripts above, you probably saw they fail. That's because they didn't find a web application running at http://localhost/login or because the application was found but it did not match the expected behavior. You may adapt your application and run the tests again.
Concordia shows a report that indicates the failures' locations. They help you to decide if a failure was caused by the application under test (e.g., it did not behave as expected) or because of the requirements specification (e.g., it is outdated in relation to the application).
Now keeping your specification updated has a new clear benefit: you can use it to generate tests and discover existing defects in your application!