Overview of the Concordia declarations:

Features and Scenarios

A Featuredefines a desired behavior for the software, a piece of functionality that gives some business value. A Scenario is a high-level usage scenario of a Feature. A Feature can have many Scenarios.

Since Feature and Scenario are business-oriented declarations, they are not used by Concordia Compiler to generate test cases. You can write them freely. Although, we recommend you to use the User Story template for a Feature and the Given-When-Then template for a Scenario. Example:

Feature: Sales Report
  As a manager
  I would like to generate a sales report
  In order to track company sales

Scenario: Daily sales report
  Given that I am authenticated as a manager
  When I open the sales report
    and I choose the option "Daily report"
  Then I see a list with all the today's sales 
    and the total value

This is the User Story template:

• As a <role>

• I would like to <goal to perform>

• In order to <received benefit> (or So that <received benefit>)

And this is the Given-When-Then (GWT) template:

• Given <some context or precondition>

• When <some action is carried out>

• Then <a result or postcondition is observed>

In the GWT template, additional sentences of each kind are written in the next line and receive the prefix and.

Features and Scenarios must have a name, but their description are not required. Thus, you can write the above example like this (although we recommend against):

Feature: Sales Report

Scenario: Daily sales report

Feature files receive the extension .feature. Only one feature per file is allowed.

Variants

A Variant is a test-oriented declaration that describes a possible interaction between a user and the system in order to accomplish a Scenario. You can write more than one Variantfor a same Scenario .

A Variant must use the GWT template and first person singular ("I") in its sentences. Concordia Compiler uses Natural Language Processing (NLP) techniques to understand Variant sentences, aiming to transform them into test cases. Example:

Feature: New User

Scenario: Successful registration

  Variant: Register with a new account
    Given that I am on "https://example.com/register"
    When I fill <#name> with "Bob"
      and I fill <#dateOfBirth> with "11/30/1990"
      and I enter with "bob@example.com" in <#email>
      and I inform "m123456" in <#pass>    
      and I type "m123456" in <#confirmation>
      and I click <#ok>
    Then I see "Welcome, Bob" 

Concordia Compiler understands the sentences from line 7 to11 as being variations of the same action, fill.

Of course, the vocabulary is limited. Although, you can extend it whether you need (by adding new synonyms to a specific dictionary file). Actions and common variations are documented and suggestions are always welcomed.

In the sentences above, the values were denoted between quotation marks, e.g. "bob@example.com". Numeric values are also accepted and do not need quotation marks. Example: 97.

Widgets (of the user interface) are written between < and >. Example: <#email>. Concordia adopts the well-known CSS query selectors to locate widgets, regardless the user interface (its plugins convert them to the format adopted by the testing framework). You can also use XPath (although some plug-ins may not accept it). Examples:

• # (hashtag) to find a widget by its id. Example: #email

• . (dot) to find a widget by its class, if applicable. Example: .colorful

• @ (at) to find a widget by its name. Example: @pass

• ~ (tilde) to find a widget by its mobile name, if applicable. Example: ~pass

• no prefixed characters, to find a widget by its type. Example: input

• // (double slash) to find a widget by its XPath. Example: //table/tbody/tr[1]

States

A State is denoted by a text between tilde (~), such as ~a state~, and must be written inside Variant sentences only. Example:

Then I have a ~user logged in~

A Variant can both produce or require states. A State is produced by a Then sentence, like in the prior example. A State is required by a Given or a When sentence. For instance:

Given that I have ~user logged in~

By declaring a required State, you're establishing a dependency among the current Variant and the ones that can produce it. Since that State is often produced by a Variant from another Feature, you have to import the corresponding Feature file. Concordia Compiler will only search for states from imported Feature files.

When transforming Variants into Test Cases, Concordia Compiler will:

• Remove any (Then) sentences with a produced State; and

• Replace any (Given or When) sentences with a required State by sentences from the Variant that can produce it.

Whether there are different Variants that can produce a (same) required State, Concordia Compiler will be able to combine each of them with the current Variant, aiming to produce Test Cases that explore different execution paths. For more information, please read the section State Based-Combination.

Example

Let's say that we have a simple Login feature like this:

Feature: Login

Scenario: Sucessful login

  Variant: Login with username and password
    Given that I am on the [Login Screen]
    When I enter with "bob" in {Username}
      and I enter with "123456" in {Password}
      and I click on {OK}
    Then I see "Welcome"
      and I have ~user logged in~


Constants:
  - "Login Screen" is "/page"
  
UI Element: Username

UI Element: Password

UI Element: OK
  - type is button

Now let's suppose that a user has to be logged in to access its account details:

import "login.feature"

Feature: My Account

  Scenario: See my account data
  
    Variant: Show basic data by default
      Given that I have ~user logged in~
        and I am on the [Account Screen]
      Then I see {User} with "bob"
        and I see {Name} with "Robert Downey Jr"


Constants:
  - "Account Screen" is "/account"
        
UI Element: User

Concordia Compiler will produce my-account.testcase with a Test Case like the following, that includes steps from both the features:

import "my-account.feature"

@scenario(1)
@variant(1)
Test Case: See my account data - 1
  Given that I am on the "/login"
  When I enter with "bob" in <username>
    and I enter with "123456" in <password>
    and I click on <ok>
  Then I see "Welcome"
  Given that I am on the "/account
  Then I see <user> with "bob"
    and I see <name> with "Robert Downey Jr"

Test Cases

A Test Case represents a test case produced for a certain Variant . Concordia Compiler can generate many Test Case declarations from a same Variant, each one covering different states, input combinations and oracles. The generation considers the following declarations:

• Variant

• states

• UI Element

• Constants

• Table

• Database

The simplest Test Case is one whose sentences are exactly equal to the Variant's. That's the case when the Variant does not use other declarations and it explicits all the input test data to use.

For comparison, whether we simply remove with "Bob"from the sentenceWhen I fill <#name> with "Bob" , Concordia Compiler will generate a pseudo-random value to complete the sentence, and it will be like this: When I fill <#name> with "A~!39esljdns so2u1 *ns%".

Concordia Compiler uses a single seed to generate pseudo-random numbers in its algorithms and values. Everytime it runs, a new seed is produced. You can set the seed using the CLI parameter --seed. By setting the seed, the compiler will pick the same paths and produce the same values - which is desirable for reproducing the same behavior in the application under test, but not for discovering new defects.

UI Elements

A UI Element represents a user interface element and can denote its expected behavior through properties.

Although a Widget is easy to declare, it can have some shortcomings:

1) It can make a sentence hard to read, specially when it uses XPath or CSS locators.

When I click on <//*[@id="section1"]/div[1]/div[2]>

When I click on {Continue}

UI Element: Continue
  - locator: //*[@id="section1"]/div[1]/div[2]  

2. It can be hard to maintain. When a Widget appears in more than one Scenario and its locator needs to change (e.g. you have changed it in the UI), you need to search and replace all its occurrences in the document. With a UI Element, you just need to change the property locator.

3. It does not offer resources for test generation other than its locator.

A UI Element offers properties that can describe the behavior related to it. These properties are used to infer the test data and test oracles to generate.

Example:

UI Element: Net Price
  - minimum value is 10.00
    Otherwise I see "Net Price must be greater than U$ 10"    

These are the UI Element properties:

• id or locator ⇨ how to locate the widget

• type ⇨ widget type, like textbox, button, etc.

• editable ⇨ whether the UI Element is editable (or not)

• data type ⇨ data type of an editable UI Element

• required⇨ whether the UI Element is required (or not)

• format⇨ data format, denoted by a regular expression

• value⇨ defines how the value is produced

• minimum value ⇨ defines how the minimum value is produced

• maximum value ⇨ defines how the maximum value is produced

• minimum length ⇨ defines how the minimum length is produced

• maximum length ⇨ defines how the maximum length is produced

By default, declaring a UI Element without properties is the same as declaring it with:

• id or locator equal to the UI Element name in camelCase (e.g., "Net Price" becomes "netPrice"). The adopted case is configurable.

• type equal to textbox

• data type equal to string

Another example:

UI Element: Profession
  - value is in [ "Programmer", "Tester", "Analyst" ]

Please see the language reference for more details. Let's proceed with the other declarations for now.

Import

A Import declaration allows you to import declarations from another .featurefile. Example:

Import "login.feature"

Imported declarations are Feature, Constants, Table, Database, and UI Element.

Constants

A Constants block defines one or more constants. Every Constant holds an immutable value that is accessible through the declared name. Example:

Constants:
  - "Register Page" is "https://example.com/register"
  - "Coupon Number" is 12345

Constants are accessed by their name between [ and ]. Example:

    Given that I am on the [Register Page]
    When I fill {Coupon} with [Coupon Number]

Constants are global declarations and cannot have duplicate names. You can use them in UI Element properties and Variant sentences.

Table

A Table declares values in rows and columns to use in UI Element properties. Example:

Table: Customer
  | name           | age |
  | Ada Lovelace   | 36  |
  | Dennis Ritchie | 70  |
  | Donald Knuth   | 82  |
  | George Boole   | 49  |
  | Niklaus Wirth  | 83  |
  
  
UI Element: Name
  - value comes from "SELECT name FROM [Customer]"
    Otherwise I see "Customer not found."

Note that SQL is used to select values from the declared table, which is denoted between [ and ] (like Constants). Table names are global and share the same namespace as Constant names.

Database

A Database provides a way to indicate a connection to an existing database or file. Example:

Database: AcmeDB
  - type is "mysql"
  - name is "acme"
  - host is "http://127.0.0.1"
  - username is "tester"
  - password is "testing123"
  
  UI Element: Name
  - value comes from "SELECT name FROM [AcmeDB.customer]"
    Otherwise I see "Customer not found."

SQL is used to select values from a table or view from the declared database, which is denoted between [ and ] . Its table or view names are separated by a dot (.). Database names are global and share the same namespace as Tables and Constants.

Test Events

In order to set the execution environment up, testers can define events that can run database commands or command-line scripts. These events can occur before or after Scenarios or Features:

• Before Each Scenario: executes before every test of each Scenario of the current Feature.

• After Each Scenario: executes after every test of each Scenario of the current Feature.

• Before Feature: executes once, before all the tests of the current Feature.

• After Feature: executes once, after all the tests of the current Feature.

• Before All: executes once, before all the tests of the application under test.

• After All: executes once, after all the tests of the application under test.

Test Events are written like Variants, using Given-When-Then. Since they usually perform actions, it is very likely to useWhen in most sentences. Example:

Before Feature:
   Given that I connect to [AcmeDB]
   When I run the script 'DELETE FROM customer'
     and I run the script 'INSERT INTO customer ( name, age ) VALUES ( "Alice", 20 ), ( "Bob", 30 )'     
     and I run the command 'rmdir log'

Commands and database scripts are denoted between apostrophe (').

⚠ UNDER CONSTRUCTION ⚠

In software testing, test coverage is some kind of metric that helps you to understand what parts of your application (or code) are exercised by your tests. The number of feasible paths through them grows exponentially with an increase in application size and can even be infinite in the case of applications with unbounded loop iterations. That is a problem called path explosion problem. Concordia Compiler deals with it by providing sets of combination and selection strategies, and trying to achieve full path coverage over time.

Test Coverage

• All the Features, Scenarios, and Variants are covered by default.

• The CLI parameter --files can filter the .feature files to be considered.

• The CLI parameter --ignore can indicate the .feature files to be ignored, when a directory is given.

• The tag @ignore can be used to mark a Feature or Variant to be ignored by the test generator. However, it can still be used by other Features or Variants.

• The tag @importance (e.g., @importance( 8 )) can be used to denote the importance of a Feature. CLI parameters --sel-min-feature and --sel-max-feature can then be used to filter the features to be considered by the test generator. Example: concordia --sel-min-feature 7 makes the compiler considers the features with importance value of 7 or above. By default, all the features receive an importance value of 5.

• Variants are selected and combined using State-based Strategies (see below).

• All the UI Elements constraints are covered by default, using a set of Testing Techniques (see below).

State-based Strategies

In Concordia, you can declare a State in a Variant sentence using a text between tile (~), like this:

Given that I have ~payment method selected~

There are three types of State:

1. Precondition: when declared in a Given sentence;

2. State Call: when declared in a When sentence;

3. Postcondition: when declared in a Then sentence.

Both Preconditions and State Calls are considered required States. That is, they denote a dependency of a certain State of the system that needs to be executed. A Precondition needs to be executed before the Variant starts, and a State Call needs to be executed during the Variant's execution.

A Postcondition is a produced State, that is, a state of the system produced by a successful execution of a Variant. Therefore, whether something goes wrong during a Variant's execution, it will not produce the declared State.

When the current Variant requires a State, Concordia Compiler will look for imported Features' Variants able to produce it. To generate complete Test Cases for the current Variant, it will:

1. Select the Variants to combine;

2. Generate successful test scenarios for the selected Variants;

3. Select the successful test scenarios to combine;

4. Generate (successful and unsuccessful) test scenarios for the current Variant;

5. Combine the selected successful test scenarios with the test scenarios of the current Variant;

6. Transform all the test scenarios into test cases (i.e., valued test scenarios).

Steps 1 and 3 can adopt different strategies. Concordia Compiler let's you:

• Parameterize how the Variants will be selected, using --comb-variant; and

• Parameterize how the successful test scenarios will be combined, using --comb-state.

Variant selection

Available strategies for --comb-variant:

• random: Selects a random Variant that produces the required State. That's the default behavior;

• first: Selects the first Variant that produces the required State;

• fmi: Selects the first most important Variant (since two Variants can have the same importance value) that produces the required State;

• all: Selects all the Variants that produce the required State.

Example:

npx concordia --comb-variant=all

State combination

Available strategies for --comb-state:

• sre: Single random of each - that is, randomly selects a single, successful test scenario of each selected Variant. That's the default behavior;

• sow : Shuffled one-wise - that is, shuffles the successful test scenarios than uses one-wise combination.

• ow: One-wise selection;

• all: Selects all the successful test scenarios to combine.

Example:

npx concordia --comb-state=all

Full vs random selection

Strategies that use random selection can take different paths every time they are used. Furthermore, they reduce considerably the amount of generated paths - i.e., it avoids "path explosion" - and thus the amount of produced test cases.

Full-selection strategies can be used for increase path coverage. Although, it also increases the needed time to check all the paths, which may be undesirable for frequent tests.

By default, Concordia Compiler uses random selection strategies.

Testing Techniques

Concordia Compiler can infer input test data from Variants, UI Elements, Constants, Tables, and Databases. The more constraints you declare, the more test cases it generates.

Adopted techniques to generate input test data include:

• Boundary-value analysis

• Equivalence partitioning

• Random testing

• Special cases

These are well-known, effective black-box testing techniques for discovering relevant defects on applications.

Data Test Cases

We call Data Test Cases those test cases used to generate input test data. They are classified into the following groups: RANGE, LENGTH, FORMAT, SET, REQUIRED, and COMPUTED. The group COMPUTEDis not available on purpose, since a user-defined algorithm to produce test data can have bugs on itself. Thus, one should provide expected input and output values in order to check whether the application is able to correctly compute the output value based on the received input value.

Every group has a set of related data test cases, applied according to the declared constraints and selected algorithms:

Maximum length for random string values

By default, the maximum length for randomly-generated string values is 500. This value is used for reducing the time to run test scripts, since long strings take time to be entered.

You can set maximum length using the CLI parameter --random-max-string-size. Example:

npx concordia --random-max-string-size=300

You can also set it in the configuration file (.concordiarc) by adding the property randomMaxStringSize. Example:

"randomMaxStringSize": 300

Properties vs Data Test Cases

Data Test Cases (DTC) are selected for every declared UI Element and its properties. The more properties you declare for a UI Element, the more data you provide for Concordia Compiler to generate DTC.

Example of some evalutations:

• When no properties are declared, FILLED and NOT_FILLED are both applied and considered as valid values;

• When the property required is declared, NOT_FILLED (empty) is considered as an invalid value;

• When the property value is declared:

  • if it comes from a set of values (inclusing a query result), all the DTC of the group SET are applied;

  • otherwise, ...

• When the property minimum value is declared:

Example 1

Let's describe a user interface element named Salary :

UI Element: Salary
  - data type is double

When no property is defined or only the property data typeis defined,

We defined the property data type as double, since the default data type is string.

Since few restrictions were made, Salary will be tested with the test cases of the group REQUIRED:

1. FILLED: a pseudo-random double value is generated;

2. NOT_FILLED: an empty value will be used.

Now let's add a minimum value restriction.

UI Element: Salary
  - data type is double
  - minimum value is 1000.00
    Otherwise I must see "Salary must be greater than or equal to 1000"

Some tests of the group RANGE are now applicable:

1. LOWEST_VALUE: the lowest possible double is used

2. RANDOM_BELOW_MIN_VALUE: a random double before the minimum value is generated

3. JUST_BELOW_MIN_VALUE: a double just below the minimum value is used (e.g., 999.99)

4. MIN_VALUE: the minimum value is used

5. JUST_ABOVE_MIN_VALUE: a double just above the minimum value is used (e.g., 1000.01)

6. ZERO_VALUE: zero (0) is used

Since 1000.00 is the minimum value, the data produced by the tests 1, 2, 3, and 6 of the group VALUE are considered invalid, while 4 and 5 are not. For these tests considered invalid, the behavior defined in Otherwise, that is

    Otherwise I must see "Salary must be greater than or equal to 1000"

is expected to happen. In other words, this behavior serves as test oracle and must occur only when the produced value is invalid.

Unlike this example, when the expected system behavior for invalid values is not specified and a test data is considered invalid, Concordia expects that test should fail. In this case, it generates the Test Case with the tag @fail.

Now let's add maximum value restriction:

UI Element: Salary
  - data type is double
  - minimum value is 1000.00
    Otherwise I must see "Salary must be greater than or equal to 1000"
  - maximum value is 30000.00
    Otherwise I must see "Salary must be less than or equal to 30000"

All the tests of the group RANGE are now applicable. That is, the following tests will be included:

1. MEDIAN_VALUE: the median between the minimum and the maximum values

2. RANDOM_BETWEEN_MIN_MAX_VALUES: a pseudo-random double value between the minimum and the maximum values

3. JUST_BELOW_MAX_VALUE: the value just below the maximum value

4. MAX_VALUE: the maximum value

5. JUST_ABOVE_MAX_VALUE: the value just above the maximum value

6. RANDOM_ABOVE_MAX_VALUE: a pseudo-random double above the maximum value

7. GREATEST_VALUE: the greatest possible double

The tests from 5 to 7 will produce values considered invalid.

Example 2

Let's define a user interface element named Profession and a table named Professions from which the values come from:

UI Element: Profession
  - type is select
  - value comes from "SELECT name from [Professions]"
  - required

Table: Professions
  | name       |
  | Lawyer     |
  | Accountant |
  | Dentist    |
  | Professor  |
  | Mechanic   |

Applicable test are:

• FILLED

• NOT_FILLED

• FIRST_ELEMENT

• RANDOM_ELEMENT

• LAST_ELEMENT

• NOT_IN_SET

The first two tests are in the group REQUIRED. Since we declared Profession as having a required value, the test FILLED is considered valid but NOT_FILLED is not. Therefore, it is important to remember declaring required inputs accordingly.

The last four tests are in the group SET. Only the last one, NOT_IN_SET, will produce a value considered invalid.

Example 3

In this example, let's adjust the past two examples to make Salary rules dynamic and change according to the Profession.

Firstly, we add two columns the the Professions table:

Table: professions
  | name       | min_salary | max_salary |
  | Lawyer     | 100000     |  900000    |
  | Accountant |  90000     |  800000    |
  | Dentist    | 150000     |  900000    |
  | Professor  |  80000     |  500000    |
  | Mechanic   |  50000     |  120000    |

Then, we change the rules to retrieve the values from the table:

UI Element: Salary
  - data type is double
  - minimum value comes from the query "SELECT min_salary FROM [Professions] WHERE name = {Profession}"
    Otherwise I must see "The given Salary is less than the minimum value"
  - maximum value comes from the query "SELECT max_salary FROM [Professions] WHERE name = {Profession}"
    Otherwise I must see "The given Salary is greater than the maximum value"

The reference to the UI Element {Profession} inside the query, makes the rules of Salary depend on Profession. Every time a Profession is selected, the minimum value and the maximum value of Salary changes according to the columns min_salary and max_salary of the table Professions.

Concordia Compiler uses plug-ins for:

1. setting the testing environment up;

2. generating and executing test scripts;

3. converting and reporting test results.

Every plug-in can generate test scripts for a different testing framework or programming language.

Plug-ins

Notes:

• (*) RTS means "Requires a Testing Server", that is, if it requires a testing server to run the test scripts (e.g., Selenium Standalone).

• (**) Appium requires Java for testing web-based mobile applications or using the Android SDK.

• codeceptjs-testcafe uses the frameworks CodeceptJS and TestCafé, and works with probably any browser.

• codeceptjs-playwright uses the frameworks CodeceptJS and Playwright,

  requires Node 10.14 or above, and works with Chromium, Firefox, and Safari.

• codeceptjs-webdriverio uses the frameworks CodeceptJS and WebDriverIO, and works with Chrome, Firefox, IE, and Edge.

• codeceptjs-appium uses the frameworks CodeceptJS and Appium, and requires some additional configuration to work.

• codeceptjs-testcafe and codeceptjs-playwright are only available for Concordia Compiler 2 or above.

Installation

Use --plugin-install plus the plug-in name. Example:

npx concordia --plugin-install codeceptjs-testcafe

Upgrade

Just uninstall the plug-in and then install it again. Example:

npx concordia --plugin-uninstall codeceptjs-playwright
npx concordia --plugin-install codeceptjs-playwright 

How to use a plug-in

Concordia has the following plug-in commands:

• plugin-install to install a plug-in

• plugin-uninstallto uninstall a plug-in

• plugin-serve to start a testing server using the plug-in

• plugin to use a plug-in

• plugin-info to show information about a plug-in

• plugin-list to list installed plug-ins

All but the latter command (plugin-list) require a plug-in name.

👉 Whether you have a configuration file with the property plugin defined, you can omit the plugin name from a command. Example:

{
  "plugin": "codeceptjs-playwright"
}

Commonly used commands

Starting a testing server

Some plug-ins (WebDriverIO, Appium) require a testing server to execute test scripts. A testing server controls a browser or an emulator during tests.

We recommend to open a new terminal/console and then start the testing server:

npx concordia --plugin-serve <plugin>

The testing server will remain open. To stop it later, type Ctrl + C.

Generating and executing test scripts

npx concordia --plugin <plugin>

Generating test scripts without executing them

You can use --no-run to avoid running test scripts and use --no-result to avoid reading the last report with test results. Test scripts will be generated but not executed.

npx concordia --plugin <plugin> --no-run --no-result

Executing existing test scripts (without generating them)

You can use --no-script to avoid generating test scripts. Only existing test scripts will be executed.

npx concordia --plugin <plugin> --no-script

See also

• Configuration file

• Commands for plug-ins

• Creating a plug-in

Concordia to retrieve test data or to set the testing environment up before or after running test scripts.

Installation

Installing

Use --db-install plus the driver name. Example:

Available driver names:

Database drivers are installed as development dependencies of your application.

Uninstalling

Use --db-uninstall plus the driver name. Example:

Upgrading

Just uninstall and then install it again.

Usage

A is global declaration. You must declare it once and then import its file in other features. Example:

A database can be referenced inside UI properties and Test Events by its name between [ and ]. Example: [My DB]. To reference a database table, separate the table name from the database name with with a dot (.) - e.g. [My DB.profession]. Let's see an example:

Instead of reinventing the wheel, Concordia uses SQL to query and modify databases and files. Even JSON and INI files can be queried with SQL.

In the example above, the property value from the UI Element "Profession" indicates that its (valid) values are retrieved from the table "profession" that belongs to the database declared as "My DB". The test event Before Each Scenario was declared to connect to the database, clean the table "profession", and then insert new records that the UI Element "Profession" will use.

See also

Imagine a solution in which you can write software requirements using an Agile-style, restricted natural language and it automatically generates for you using effective testing techniques such as , , and and it can also use test data from . Well, that's Concordia. Its solution is composed by a metalanguage and a compiler.

Concordia metalanguage

Concordia allows you to focus on writing your application's business rules and defining its expected behavior for when users interact with it. Stakeholders, business analysts, testers, and programmers can now discuss the application using a single language, a single source of truth. Business-focused and testing-focused declarations are clearly separated, so you can discuss technological details only with the interested parties.

With Concordia, you can write both and requirements, although it produces only functional test cases. Because it adopts -style declarations, you don't have to write it so formally (like usually do) and you can use plain text - which is easy to evolve with your codebase.

We invite you to and see how it is easy to understand. It is currently available in English (en) and Portuguese (pt).

Concordia compiler

Concordia compiler mix techniques with and to understand declarations in Concordia Language - which uses restricted natural language. After detecting declarations, it infers test scenarios, test data, and test oracles to generate test cases in a framework-independent format (see the ). Finally, it uses a plug-in created for a specific testing framework to generate test scripts (that is, source code), execute them and read their results. You don't need to write code. And the compiler checks your declarations for logic errors and other possible problems.

Follow the to see it in action! 😉

Concordia plug-ins

Concordia compiler uses plug-ins to transform Abstract Test Cases (ATS) into test scripts - that is, into source code. Every plug-in can be implemented to generate source code for a specific programming language and testing framework - in Java, Python, Go, PHP or whatever the language you prefer. The generated source code is not tied to JavaScript or TypeScript.

See the and for your favorite programming language and testing framework.

Tools

• is a browser extension for Chrome and Firefox that converts recordings from (which is a - that is, it records all the actions that you perform while interacting with the application under test in order the reproduce them later) to Variant sentences. That's very handful for discovering elements' identification in web applications (e.g., their id properties or their ) and obtaining Variant sentences with the corresponding interaction.

• is a command-line tool that generates User Interface Prototypes (UIP) from Concordia features. It can help discussing features with stakeholders and accelerating their construction. Currently there is a customizable plug-in for HTML-based interfaces.

• is a command-line tool that generates an HTML page with an interactive graph that shows features files' relationships.

Publications

1. PINTO, Thiago Delgado. Unifying Agile requirements specification quality control and implementation conformance assurance. 2018. Doctoral Thesis. Pontifical Catholic University of Rio de Janeiro, Informatics Department, 2018.

2. PINTO, Thiago Delgado; COSTA, Pablo Veiga; GONÇALVES, Willian Inácio. User Interface Prototype Generation from Agile Requirements Specifications written in Concordia. 2019. WebMedia'19: Proceedings on the 25th Brazilian Symposium on Multimedia and the Web, Rio de Janeiro, 2019. ACM. DOI:

Concordia Language contains a set of meta-actions that can be used by different testing frameworks for generating test scripts. Concordia Compiler uses Intent Recognition (yes, like chatbots do) to understand sentences and extract the desired action to perform. Later these actions are sent to testing frameworks for conversion into commands.

The following list presents the available actions and gives some examples on how to use them in sentences. We expect that you do not have to memorize them but, instead, see the things you can do. Try to write the sentences the way you speak, using first person singular (I) . When there is a UI Element in the sentence, embrace it with { and }, like {this}. The compiler will tell you whether it cannot understand some sentence you wrote. If you think it should have understood something, . We'll try to augment the compiler's vocabulary to understand it the next time.

accept

Accepts a browser message or app message.

amOn

Indicates a webpage or screen in which it is expected to be at.

append

Adds a value to an element.

attachFile

Attaches a file. It selects a file and confirms its choice (e.g., clicks Open).

cancel

Cancels a browser message or app message.

check

Checks a checkbox.

clear

Empties an input field or browser cookie

click

Clicks on something in the screen.

close

Closes a tab, a set of tabs, or an app (mobile only).

connect

Connects to a database.

The next sentence is for only:

disconnect

Disconnects from a database.

The next sentence is for only:

doubleClick

Performs a double click on something.

drag

Drags and drops something to a widget.

fill

Indicates that a field is being filled. If desired, a value can be given. Otherwise, a value will be generated for the corresponding Test Case.

fill + target

hide

Hides something.

install

Installs an app.

maximize

Maximizes a window or the browser.

move

Moves the mouse cursor to a place or element.

open

Opens something

press

Presses a key or key combination, separated by comma.

Some special keys (case sensitive!):

• Add

• Alt

• ArrowDown or Down arrow

• ArrowLeft or Left arrow

• ArrowRight or Right arrow

• ArrowUp or Up arrow

• Backspace

• Command

• Control

• Del

• Divide

• End

• Enter

• Equals

• Escape

• F1 to F12

• Home

• Insert

• Meta

• Multiply

• Numpad 0 to Numpad 9

• Pause

• Pagedown or PageDown

• Pageup or PageUp

• Semicolon

• Shift

• Space

• Subtract

• Tab

pull

Extracts a device's resource from a path.

refresh

Refreshes/reloads the current page.

remove

Removes/uninstall an app by its internal name (mobile only).

resize

Resizes a window.

rightClick

Performs a right click on something.

run

Runs a console command or a database script (SQL command).

👉 Available for only.

👉 Commands and SQL scripts must be declared between single quotes ('), in a single line.

👉 SQL scripts must reference their database.

SQL Syntax

Concordia Compiler uses to access databases and files through SQL-like commands. The supported SQL syntax may vary from one database to another. Please see the .

JSON and CSV as databases: INSERT accepts JSON objects or arrays as values. Example:

Excel and Firebase databases: Syntax similar to JSON and CSV databases. However, it has some limitations, as pointed out in :

SQL commands are limited to SELECT, UPDATE, INSERT and DELETE. WHERE works well. JOINs are not allowed. GROUP BY is not supported. LIMIT and OFFSET are combined into a single LIMIT syntax: LIMIT [offset,]number

INI databases: INSERT and DELETE are not supported yet by . UPDATE example:

SQLite databases: Currently uses that doesn't persist the changes in the file (only in memory).

saveScreenshot

Takes a screenshot an saves into a file.

scrollTo

Scrolls to a certain element.

see

Indicates that something can be seen. You can also negate the sentence to indicate something cannot be seen.

select

Selects a value for an element.

shake

Shakes the device - MOBILE ONLY.

swipe

Performs a swipe action - MOBILE ONLY

switch

• Switches to a certain iframe and back to the application page; OR

• Switches to a certain tab; OR

• Switches an app to native mode or web mode - MOBILE ONLY.

NOTE: When switching to an iframe, all the commands that follow it will be applied to it.

tap

Performs a tap on an element - MOBILE ONLY

uncheck

Unchecks a checkbox.

wait

Wait for something.

Concordia Compiler can use a configuration file in JSON format named .concordiarc. You can generate it by running the following command:

npx concordia --init

You'll be asked about the desired configuration and then the file will be saved.

Example:

{
    "language": "pt",
    "directory": "docs/features",
    "plugin": "codeceptjs-testcafe",
    "dirScript": "test/e2e",
    "dirResult": "test/e2e/output"
}

Configuration file's properties are similar to CLI parameters. The example above defines: "pt" (meaning "Portuguese") as the default language; the directory "docs/features" for features and test cases; the plug-in "codeceptjs-testcafe" for generating test scripts; the directory "test/e2e" for test scripts; and the directory "test/e2e/output" for output files such as report files and screenshots.

File-related options

directory

Directory to search for .feature and .testcase files.

Example:

{
  "directory": "./features"
}

recursive

Use recursive directory search.

Example:

{
  "recursive": false
}

files

Files to consider, instead of considering directory. The files must be separated by colon.

Example:

{
  "files": "/path/to/file1.feature,/other/file2.feature"
}

Another example:

{
  "files": [
    "/path/to/file1.feature",
    "/other/file2.feature"
  ]
}

ignore

Files to ignore, considering the parameter directory. The files must be separated by colon. Example:

{
  "ignore": [
    "/path/to/file1.feature",
    "/other/file2.feature"
  ]
}

extensions

To-Do

encoding

File encoding of the files. Accepted values are:

• utf8 or utf-8

• latin1

• ascii

• ucs2 or ucs-2

• utf16le or utf-16le

Naturally, the files must be edited with the given file encoding.

Language-related options

To-Do

Plug-in related options

To-Do

Processing-related options

To-Do

Generation-related options

To-Do

Strategy-related options

To-Do

Basic syntax is:

concordia [dir] [options]

where:

• A parameter between[ and ] means that it is optional.

• [dir] is the directory that contains your .feature files. Whether it is not informed, the tool searches all the subdirectories from the current path.

• [options] changes how the tool behaves.

You can run concordia --help to see all the command line interface (CLI) options.

Configuration

--config

Indicates a configuration file

Alias: -c

concordia --config /path/to/.concordiarc

By default the configuration file is loaded from the current directory.

--init

Creates a configuration file and optionally installs plug-ins and databases.

concordia --init

This command asks about your preferences. By hitting Enter without giving an answer, it will select the suggested (default) value. You can use Up and Down to navigate the answers and hit Enter to select it. For multiple selection (when available), hitSpace.

By using a configuration file (e.g. .concordiarc) you avoid informing the configured parameters in the console/terminal. For instance, whether plugin is defined in the configuration file, you don't need to inform --plugin anymore.

--save-config

Add CLI parameters to the configuration file

The following example will consider the given directory and add it to the configuration file:

concordia --save-config --directory ./features

Directories and files

--directory

Indicates the directory that has .feature and .testcase files.

Alias: -d

When not given, it will search the current directory and all subdirectories recursively.

Example:

concordia --directory ./features

This is equivalent to:

concordia ./features

--file

Specifies .feature files to consider

Aliases: -f, --files

Files must be separated by comma.

concordia --file "file1.feature,/path/to/file2.feature"

Whether the command also informs a directory (e.g. --directory), it searches the files inside the directory. Example:

concordia ./feature --file "f1.feature,f2.feature"

The example above will search for ./feature/f1.feature and ./feature/f1.feature.

--ignore

Specify .feature files to ignore when a directory is informed

Alias: -i

Files must be separated by comma.

concordia ./feature --ignore "f3.feature,f4.feature"

--no-recursive

Disables recursive search when a directory is informed

concordia ./features --no-recursive

Output directories

--dir-result

Specifies the directory for test reports and screenshots.

Alias: -O

concordia --dir-result output

--dir-script

Specifies the directory for test script files.

Alias: -o

concordia --dir-script test

Language and locale

--language

Sets the default language to consider.

Alias: -l

The language code must have only 2 characters and needs to be one of the available languages.

concordia --language pt

--language-list

Lists the available language codes.

concordia --language-list

--locale-list

Lists the available locales.

concordia --locale-list

Processing and output

--just-script

Just generates test scripts

concordia --just-script

--just-spec

Just verifies specification files

concordia --just-spec

--just-test-case

Just generates test cases

concordia --just-script

--just-run

Just execute test script files

concordia --just-run

--no-script

Avoids generating test scripts

concordia --no-script

--no-spec

Avoids processing specification files

concordia --no-spec

--no-test-case

Avoids generating test cases

concordia --no-test-case

--no-result

Avoids reading test scripts' results

concordia --no-result

--no-run

Avoids running test scripts

concordia --no-run

concordia -x

--verbose

Uses a verbose output

concordia --verbose

Test script execution

--instances

Number of parallel instances to execute.

Alias: -I

concordia --instances 2

--headless

Executes browsers in headless mode.

Alias: -H

concordia --headless

--script-file

Sets one or more test script files to execute

Aliases: -F, --script-files

Files must be separated by comma.

concordia --script-file "test1.js,test2.js"

--script-grep

Sets a text or regular expression to filter the files to execute

Alias: -G

concordia --script-grep "Sales"

--target

Sets target browsers or platforms to execute

Alias: -T

Browser or platforms must be separated by comma.

concordia --target "chrome,firefox,edge"

Plug-in

--plugin

Uses the given plug-in (if installed)

Alias: -p

concordia --plugin <plugin>

concordia -p <plugin> -x

--plugin-about

Shows information about an installed plug-in

Alias: --plugin-info

concordia --plugin-info <plugin>

--plugin-install

Installs a plug-in

concordia --plugin-install <plugin>

--plugin-list

Lists installed plug-ins

concordia --plugin-list

--plugin-serve

Starts a testing server for a plug-in

Alias: -S

concordia --plugin-serve <plugin>

The testing server will keep running. Thus, it's a good idea to open a separate terminal/console for it. To stop it later, type Ctrl + C.

--plugin-uninstall

Uninstalls a plug-in

concordia --plugin-uninstall <plugin>

Database

--db-install

Installs a database driver

concordia --db-install <database>

--db-list

Lists installed databases

concordia --db-list

--db-uninstall

Uninstalls a database driver

concordia --db-uninstall <database>

Random generation

--random-max-string-size

Maximum random string size to generate, when free values are allowed.

The default value is 500 .

concordia --random-max-string-size 100

--random-min-string-size

Minimum random string size to generate, when free values are allowed.

The default value is 0 .

concordia --random-min-string-size 1

--random-tries

Random tries to generate invalid values.

When there is a set of valid values and any value different from those in the set is considered an invalid, random generation is used to produce an invalid value. This parameter defines how many tries to take. Usually it gets on the first try.

The default value is 5 .

concordia --random-tries 10

--seed

Sets a random seed

concordia --seed="example"

Concordia Compiler can use random selection in its algorithms. By default, it generates the seed using the current date and time. Using the same seed makes the algorithms produce the same output (e.g., same values and paths). While that's interesting for reproducing a past behavior, we don't recommend it for finding new defects. Using new seeds, Concordia Compiler will select different test data, test oracles, and test paths over time.

Combination strategies

--comb-data

--comb-invalid

--comb-state

--comb-variant

File formats and extensions

--line-breaker

Sets the character(s) used for breaking lines.

Since by default the line breaker is detected automatically, this command may not be useful for you in most cases.

concordia --line-breaker "\n"

Note on line breakers:

• Linux uses "\n"

• MacOS uses "\r"

• Windows uses "\r\n"

--encoding

Sets the expected file encoding

Alias: -e

The default is utf8. Possible values are:

• ascii

• latin1

• ucs2 or ucs-2

• utf16le or utf-16le

• utf8 or utf-8

concordia --encoding latin1

Information

--about

Shows information about the compiler.

concordia --about

--help

Shows the available commands

concordia --help

--newer

Verify if there is a newer version.

concordia --newer

--version

Shows the current version.

concordia --version

Migrations are only needed when upgrading to a major version. See Versions and Upgrade for more information.

To upgrade to a major version:

1. Uninstall the current version, e.g.: npm uninstall -D concordialang

2. Install the new version, e.g.: npm install -D concordialang

3. Now proceed as described below (depending on your current version).

Note: For a global installation, replace -D with -g.

From 1.x to 2.x

1. Update your plug-in

Concordia 2 has a new plugin API, not compatible with plug-ins for Concordia 1. So uninstall your current plug-in and install a new one. Example:

npx concordia --plugin-uninstall codeceptjs-webdriverio
npx concordia --plugin-install codeceptjs-webdriverio

Alternatively, you can install one of the new plug-ins then change your configuration file to use it. Example:

npx concordia --plugin-uninstall codeceptjs-webdriverio
npx concordia --plugin-install codeceptjs-playwright
npx concordia --plugin codeceptjs-playwright --save-config

2. Install database drivers

Whether your Concordia specification access a database, you have to install the corresponding database driver. Concordia 2 does not install them by default anymore, as Concordia 1 did, aiming to reduce the amount of downloaded dependencies.

Concordia 2 has a new CLI option --db-install to help with that. Example:

npx concordia --db-install mysql

See Database Drivers to install the proper driver(s) for your application.

From 0.x to 1.x

1. Update your configuration file, if needed

Whether you project has a configuration file .concordiarc, open it with a text editor. If the file has a property "plugin" with the value "codeceptjs", you must change it to "codeceptjs-webdriverio".

2. Install the new plug-in

You can install any of the available plug-ins, currently codeceptjs-webdriverio or codeceptjs-appium. Example:

 concordia --plugin-install codeceptjs-webdriverio

See also

• Information about the breaking changes

• Concordia's version numbering

Versioning

Concordia Compiler's versions are based on Semantic Versioning.

Although Semantic Versioning is conceived for APIs instead of for applications, we adopt a very similar convention. Thus, changes become predictable and you can know, from the version numbers, when a version is no more compatible with a previous version.

Given a version MAJOR.MINOR.UPDATE:

• MAIOR is increased when the Compiler or the Language is no more compatible with the previous version.

• MINOR is increased when adding functionality in a backwards-compatible manner.

• UPDATE is increased when there are fixes, little changes or little novelties - all of them backwards-compatible.

Examples:

• 0.2.0 is compatible with 0.1.0

• 0.1.1 is compatible with 0.1.0

• 1.0.0 is not compatible with 0.2.0

Upgrade

To upgrade a local installation:

npm upgrade concordialang

To upgrade a global installation:

npm upgrade -g concordialang

This page presents Concordia actions and their corresponding commands for some frameworks.

Help us improving this page. Send us a message on Slack or open an Issue at GitHub. 💖

👉 Under construction.

Information about breaking changes.

From 1.x to 2.x

Please read Issue #56.

From 0.x to 1.x

Concordia Compiler

1. It changed the way that all the plug-in operations are handled. See Issue #34 for details.

2. The behavior of the following plug-in commands were affected:

👉 See the Command Documentation to know the commands' syntax.

Concordia Language

No compatibility breaks.

Other relevant changes

External tools used by plug-ins are now installed locally, per project, instead of installed globally. Their direct execution (without using Concordia) is now possible through NPX. For instance, the command codeceptjs run test.js must now be executed as npx codeceptjs run test.js. This change also allows the installation of different versions of external tools, when needed.

FAQ

1. Were there any changes in commands' syntaxes?

   No, there were not.

2. Is my configuration file still compatible?

   It depends. All the properties are still compatible, except for "plugin". Whether your configuration file has that property, you should update it.

3. Is it possible now to install or uninstall a plug-in with NPM ?

   Yes, it is possible to do both now.

Work in progress and issues:

• Kanban Board (GitHub)

• Issues (GitHub)

Logo

Yes, this project does not have a logo yet 😱. Designers interested in helping us (for free 🙏), please contact us on our Slack Channel or open an Issue.

The logo should reflect the idea behind the language:

The tool generates test cases and test scripts that verify if the application under test meets the specified functional requirements. Therefore, the idea of "verification" or "testing" could be in the logo.

Plug-ins

Interested? See how to create a plug-in.

👉 Our next goal is a plugin for Cypress.

Suggestions:

• JavaScript and friends: Cypress (web), MacacaJS (web, mobile, desktop)

• PHP: Codeception (web)

• Java: Selenium (web), Robotium (android), AssertJ Swing (desktop)

• Ruby: Capybara (web), Calabash (mobile applications developed with Xamarin)

• Python: Selenium (web)

• C++: QtTest (desktop), DUnit (desktop)

• Delphi: DUnitX (desktop), DUnit (desktop)

• Other programming languages or frameworks are welcome! 💖