Overview of the Language

Overview of the Concordia declarations:

Features and Scenarios

A Featuredefines a desired behavior for the software, a piece of functionality that gives some business value. A Scenario is a high-level usage scenario of a Feature. A Feature can have many Scenarios.

Since Feature and Scenario are business-oriented declarations, they are not used by Concordia Compiler to generate test cases. You can write them freely. Although, we recommend you to use the User Story template for a Feature and the Given-When-Then template for a Scenario. Example:

Feature: Sales Report
  As a manager
  I would like to generate a sales report
  In order to track company sales

Scenario: Daily sales report
  Given that I am authenticated as a manager
  When I open the sales report
    and I choose the option "Daily report"
  Then I see a list with all the today's sales 
    and the total value

This is the User Story template:

• As a <role>

• I would like to <goal to perform>

• In order to <received benefit> (or So that <received benefit>)

And this is the Given-When-Then (GWT) template:

• Given <some context or precondition>

• When <some action is carried out>

• Then <a result or postcondition is observed>

In the GWT template, additional sentences of each kind are written in the next line and receive the prefix and.

Features and Scenarios must have a name, but their description are not required. Thus, you can write the above example like this (although we recommend against):

Feature: Sales Report

Scenario: Daily sales report

Feature files receive the extension .feature. Only one feature per file is allowed.

Variants

A Variant is a test-oriented declaration that describes a possible interaction between a user and the system in order to accomplish a Scenario. You can write more than one Variantfor a same Scenario .

A Variant must use the GWT template and first person singular ("I") in its sentences. Concordia Compiler uses Natural Language Processing (NLP) techniques to understand Variant sentences, aiming to transform them into test cases. Example:

Feature: New User

Scenario: Successful registration

  Variant: Register with a new account
    Given that I am on "https://example.com/register"
    When I fill <#name> with "Bob"
      and I fill <#dateOfBirth> with "11/30/1990"
      and I enter with "[e-mail]" in <#email>
      and I inform "m123456" in <#pass>    
      and I type "m123456" in <#confirmation>
      and I click <#ok>
    Then I see "Welcome, Bob" 

Concordia Compiler understands the sentences from line 7 to11 as being variations of the same action, fill.

Of course, the vocabulary is limited. Although, you can extend it whether you need (by adding new synonyms to a specific dictionary file). Actions and common variations are documented and suggestions are always welcomed.

In the sentences above, the values were denoted between quotation marks, e.g. "[e-mail]". Numeric values are also accepted and do not need quotation marks. Example: 97.

Widgets (of the user interface) are written between < and >. Example: <#email>. Concordia adopts the well-known CSS query selectors to locate widgets, regardless the user interface (its plugins convert them to the format adopted by the testing framework). You can also use XPath (although some plug-ins may not accept it). Examples:

• # (hashtag) to find a widget by its id. Example: #email

• . (dot) to find a widget by its class, if applicable. Example: .colorful

• @ (at) to find a widget by its name. Example: @pass

• ~ (tilde) to find a widget by its mobile name, if applicable. Example: ~pass

• no prefixed characters, to find a widget by its type. Example: input

• // (double slash) to find a widget by its XPath. Example: //table/tbody/tr[1]

Whether you are testing a web application, you may find Katalon-Concordia useful. It's a browser extension - available for Google Chrome and Mozilla Firefox - that extends Katalon Recorder to save a recorded interaction as Variant sentences. Using it may reduce the time needed to locate the widgets of an existing application.

States

A State is denoted by a text between tilde (~), such as ~a state~, and must be written inside Variant sentences only. Example:

Then I have a ~user logged in~

A Variant can both produce or require states. A State is produced by a Then sentence, like in the prior example. A State is required by a Given or a When sentence. For instance:

Given that I have ~user logged in~

By declaring a required State, you're establishing a dependency among the current Variant and the ones that can produce it. Since that State is often produced by a Variant from another Feature, you have to import the corresponding Feature file. Concordia Compiler will only search for states from imported Feature files.

When transforming Variants into Test Cases, Concordia Compiler will:

• Remove any (Then) sentences with a produced State; and

• Replace any (Given or When) sentences with a required State by sentences from the Variant that can produce it.

Whether there are different Variants that can produce a (same) required State, Concordia Compiler will be able to combine each of them with the current Variant, aiming to produce Test Cases that explore different execution paths. For more information, please read the section State Based-Combination.

Example

Let's say that we have a simple Login feature like this:

login.feature

Feature: Login

Scenario: Sucessful login

  Variant: Login with username and password
    Given that I am on the [Login Screen]
    When I enter with "bob" in {Username}
      and I enter with "123456" in {Password}
      and I click on {OK}
    Then I see "Welcome"
      and I have ~user logged in~


Constants:
  - "Login Screen" is "/page"
  
UI Element: Username

UI Element: Password

UI Element: OK
  - type is button

Now let's suppose that a user has to be logged in to access its account details:

my-account.feature

import "login.feature"

Feature: My Account

  Scenario: See my account data
  
    Variant: Show basic data by default
      Given that I have ~user logged in~
        and I am on the [Account Screen]
      Then I see {User} with "bob"
        and I see {Name} with "Robert Downey Jr"


Constants:
  - "Account Screen" is "/account"
        
UI Element: User

Concordia Compiler will produce my-account.testcase with a Test Case like the following, that includes steps from both the features:

import "my-account.feature"

@scenario(1)
@variant(1)
Test Case: See my account data - 1
  Given that I am on the "/login"
  When I enter with "bob" in <username>
    and I enter with "123456" in <password>
    and I click on <ok>
  Then I see "Welcome"
  Given that I am on the "/account
  Then I see <user> with "bob"
    and I see <name> with "Robert Downey Jr"

Concordia Compiler can optimize the combination among Features in order to reduce the amount of produced test cases and, therefore, the total execution time. You can also opt to get full coverage - which is ideal before releasing a feature.

Test Cases

A Test Case represents a test case produced for a certain Variant . Concordia Compiler can generate many Test Case declarations from a same Variant, each one covering different states, input combinations and oracles. The generation considers the following declarations:

• Variant

• states

• UI Element

• Constants

• Table

• Database

The simplest Test Case is one whose sentences are exactly equal to the Variant's. That's the case when the Variant does not use other declarations and it explicits all the input test data to use.

For comparison, whether we simply remove with "Bob"from the sentenceWhen I fill <#name> with "Bob" , Concordia Compiler will generate a pseudo-random value to complete the sentence, and it will be like this: When I fill <#name> with "A~!39esljdns so2u1 *ns%".

Concordia Compiler uses a single seed to generate pseudo-random numbers in its algorithms and values. Everytime it runs, a new seed is produced. You can set the seed using the CLI parameter --seed. By setting the seed, the compiler will pick the same paths and produce the same values - which is desirable for reproducing the same behavior in the application under test, but not for discovering new defects.

Don't worry about writing Test Cases. Write Variants and let Concordia generates the Test Cases for you.

UI Elements

A UI Element represents a user interface element and can denote its expected behavior through properties.

Although a Widget is easy to declare, it can have some shortcomings:

1) It can make a sentence hard to read, specially when it uses XPath or CSS locators.

For example, the sentence

When I click on <//*[@id="section1"]/div[1]/div[2]>

does inform what the action really means. The same sentence with a UI Element reference becomes

When I click on {Continue}

Then the UI Element can be declared as:

UI Element: Continue
  - locator: //*[@id="section1"]/div[1]/div[2]  

Here locator is a property that denotes how to find it in the UI.

2. It can be hard to maintain. When a Widget appears in more than one Scenario and its locator needs to change (e.g. you have changed it in the UI), you need to search and replace all its occurrences in the document. With a UI Element, you just need to change the property locator.

3. It does not offer resources for test generation other than its locator.

A UI Element offers properties that can describe the behavior related to it. These properties are used to infer the test data and test oracles to generate.

Example:

UI Element: Net Price
  - minimum value is 10.00
    Otherwise I see "Net Price must be greater than U$ 10"    

Concordia Compiler infers 6 possible test cases from the above declaration, applying different testing techniques.

These are the UI Element properties:

• id or locator ⇨ how to locate the widget

• type ⇨ widget type, like textbox, button, etc.

• editable ⇨ whether the UI Element is editable (or not)

• data type ⇨ data type of an editable UI Element

• required⇨ whether the UI Element is required (or not)

• format⇨ data format, denoted by a regular expression

• value⇨ defines how the value is produced

• minimum value ⇨ defines how the minimum value is produced

• maximum value ⇨ defines how the maximum value is produced

• minimum length ⇨ defines how the minimum length is produced

• maximum length ⇨ defines how the maximum length is produced

By default, declaring a UI Element without properties is the same as declaring it with:

• id or locator equal to the UI Element name in camelCase (e.g., "Net Price" becomes "netPrice"). The adopted case is configurable.

• type equal to textbox

• data type equal to string

Concordia Compiler is able to guess some properties from the others. For instance, editable is enabled when type is a widget capable of receiving input data; data type can be inferred from any attributed value; etc.

Another example:

UI Element: Profession
  - value is in [ "Programmer", "Tester", "Analyst" ]

Concordia Compiler infers 4 possible test cases from the above declaration, applying different testing techniques.

Please see the language reference for more details. Let's proceed with the other declarations for now.

Import

A Import declaration allows you to import declarations from another .featurefile. Example:

Import "login.feature"

Imported declarations are Feature, Constants, Table, Database, and UI Element.

Constants

A Constants block defines one or more constants. Every Constant holds an immutable value that is accessible through the declared name. Example:

Constants:
  - "Register Page" is "https://example.com/register"
  - "Coupon Number" is 12345

Constants are accessed by their name between [ and ]. Example:

    Given that I am on the [Register Page]
    When I fill {Coupon} with [Coupon Number]

Constants are global declarations and cannot have duplicate names. You can use them in UI Element properties and Variant sentences.

Table

A Table declares values in rows and columns to use in UI Element properties. Example:

Table: Customer
  | name           | age |
  | Ada Lovelace   | 36  |
  | Dennis Ritchie | 70  |
  | Donald Knuth   | 82  |
  | George Boole   | 49  |
  | Niklaus Wirth  | 83  |
  
  
UI Element: Name
  - value comes from "SELECT name FROM [Customer]"
    Otherwise I see "Customer not found."

Concordia Compiler infers 4 possible test cases from the above declaration, applying different testing techniques.

Note that SQL is used to select values from the declared table, which is denoted between [ and ] (like Constants). Table names are global and share the same namespace as Constant names.

Database

A Database provides a way to indicate a connection to an existing database or file. Example:

Database: AcmeDB
  - type is "mysql"
  - name is "acme"
  - host is "http://127.0.0.1"
  - username is "tester"
  - password is "testing123"
  
  UI Element: Name
  - value comes from "SELECT name FROM [AcmeDB.customer]"
    Otherwise I see "Customer not found."

Concordia Compiler infers 4 possible test cases from the above declaration, applying different testing techniques.

SQL is used to select values from a table or view from the declared database, which is denoted between [ and ] . Its table or view names are separated by a dot (.). Database names are global and share the same namespace as Tables and Constants.

Test Events

In order to set the execution environment up, testers can define events that can run database commands or command-line scripts. These events can occur before or after Scenarios or Features:

• Before Each Scenario: executes before every test of each Scenario of the current Feature.

• After Each Scenario: executes after every test of each Scenario of the current Feature.

• Before Feature: executes once, before all the tests of the current Feature.

• After Feature: executes once, after all the tests of the current Feature.

• Before All: executes once, before all the tests of the application under test.

• After All: executes once, after all the tests of the application under test.

Test Events are written like Variants, using Given-When-Then. Since they usually perform actions, it is very likely to useWhen in most sentences. Example:

Before Feature:
   Given that I connect to [AcmeDB]
   When I run the script 'DELETE FROM customer'
     and I run the script 'INSERT INTO customer ( name, age ) VALUES ( "Alice", 20 ), ( "Bob", 30 )'     
     and I run the command 'rmdir log'

Commands and database scripts are denoted between apostrophe (').