Welcome to the Concordia 2 documentation! This page will help you to get started fast. If you run into problems, you can find help on our Slack channel.

System Requirements

Concordia Compiler requires only version 12.20 (or above) and works on Linux, MacOS and Windows.

In this Quick Start, we'll create a simple specification of an interaction with a search page and use . Therefore, download it before continuing.

Step 0: Initialize

If you are starting a new project or whether your project does not have a file named package.json, then run:

Step 1: Install

Install Concordia Compiler :

Step 2: Configure

Concordia Compiler will ask you about some preferences. Press Enter to answer every question with their default values. It will create a configuration file named .concordiarc and install the selected and .

Step 3: Create a feature

Create the folderfeatures :

Now create the file search.feature withinfeatures, with the following content:

About the file:

• Feature and Scenario are high-level, business-focused descriptions about the problem to solve. Their sentences are not used to generate test cases. The above example does not describe them.

• A Variant describes the expected interaction with the application's user interface (UI) in order to perform a Scenario. Thus, a Variant uses technological vocabulary.

• In Concordia, all the interactions with the UI use first person singular ("I"). That "I" represents the actor that is interacting with the application (in the example above, a

Step 4: Execute

Finally, run

It will:

• set the testing environment up (once);

• generate a test case file and transformed it into a test script file;

• execute the test script file; and

Your browser should open automatically during this process and the console will report the execution results.

Your directory should now look like this:

The directory node_modules contains installed dependencies, like tools and frameworks, whose versions are managed with package.json and package-lock.json. The file codecept.json has a basic configuration to run CodeceptJS with Google Chrome and you can to fit your needs.

The file features/search.testcase should look like this:

The Test Case above was produced from the Variant declared in features/search.feature. Some notes about it:

• The import clause (line 5) imports the declared file's content.

• The tag @generated (line 7) indicates that the Test Case was produced automatically.

• The tag @scenario(1)

The file test/search.js contains the test script produced from features/search.testcase using the selected plug-in. It also contains line comments with references to their corresponding lines and columns in the .testcase file:

Notes on installation options

You can opt to install Concordia Compiler locally (per project) or globally.

Local installation (recommended)

• does not require administrative privileges (i.e., using sudo on Linux or MacOS) to install;

• allows every project to have its own, independent installation;

• demands using before every command.

Follow the for a install installation.

Global installation

• requires administrative privilegies (i.e., using sudo on Linux or MacOS) to install;

• lets you execute the compiler direcly from any folder;

• needs more attention when upgrading, especially for plug-ins.

Example:

Note: On Windows, you must omit sudo.

Additional tips:

• .

• Whether Windows requires administrative privileges to install, right click Windows (Start) menu, click "Command Prompt (Admin)" or alternatively "Windows PowerShell (Admin)", and then proceed with the installation command.

• To upgrade Concordia later, change the installation command from install to upgrade

Overview of the Concordia declarations:

Features and Scenarios

A Featuredefines a desired behavior for the software, a piece of functionality that gives some business value. A Scenario is a high-level usage scenario of a Feature. A Feature can have many Scenarios.

Since Feature and Scenario are business-oriented declarations, they are not used by Concordia Compiler to generate test cases. You can write them freely. Although, we recommend you to use the template for a Feature and the template for a Scenario. Example:

This is the template:

• As a <role>

• I would like to <goal to perform>

• In order to <received benefit> (or So that <received benefit>

And this is the (GWT) template:

• Given <some context or precondition>

• When <some action is carried out>

• Then <a result or postcondition is observed>

In the GWT template, additional sentences of each kind are written in the next line and receive the prefix and.

Features and Scenarios must have a name, but their description are not required. Thus, you can write the above example like this (although we recommend against):

Feature files receive the extension .feature. Only one feature per file is allowed.

Variants

A Variant is a test-oriented declaration that describes a possible interaction between a user and the system in order to accomplish a Scenario. You can write more than one Variantfor a same Scenario .

A Variant must use the GWT template and first person singular ("I") in its sentences. Concordia Compiler uses techniques to understand Variant sentences, aiming to transform them into test cases. Example:

Concordia Compiler understands the sentences from line 7 to11 as being variations of the same , fill.

Of course, the vocabulary is limited. Although, you can extend it whether you need (by adding new synonyms to a specific dictionary file). Actions and common variations and suggestions are always .

In the sentences above, the values were denoted between quotation marks, e.g. "bob@example.com". Numeric values are also accepted and do not need quotation marks. Example: 97.

Widgets (of the user interface) are written between < and >. Example: <#email>. Concordia adopts the well-known to locate widgets, regardless the user interface (its plugins convert them to the format adopted by the testing framework). You can also use (although some plug-ins may not accept it). Examples:

• # (hashtag) to find a widget by its id. Example: #email

• . (dot) to find a widget by its class, if applicable. Example: .colorful

States

A State is denoted by a text between tilde (~), such as ~a state~, and must be written inside sentences only. Example:

A can both produce or require states. A State is produced by a Then sentence, like in the prior example. A State is required by a Given or a When sentence. For instance:

By declaring a required State, you're establishing a dependency among the current and the ones that can produce it. Since that State is often produced by a Variant from another Feature, you have to the corresponding Feature file. Concordia Compiler will only search for states from imported Feature files.

When transforming Variants into , Concordia Compiler will:

• Remove any (Then) sentences with a produced State; and

• Replace any (Given or When) sentences with a required State by sentences from the Variant that can produce it.

Whether there are different s that can produce a (same) required State, Concordia Compiler will be able to combine each of them with the current Variant, aiming to produce that explore different execution paths. For more information, please read the section .

Example

Let's say that we have a simple Login feature like this:

Now let's suppose that a user has to be logged in to access its account details:

Concordia Compiler will produce my-account.testcase with a like the following, that includes steps from both the features:

Test Cases

A Test Case represents a test case produced for a certain Variant . Concordia Compiler can generate many Test Case declarations from a same Variant, each one covering different states, input combinations and oracles. The generation considers the following declarations:

• Variant

• states

• UI Element

The simplest Test Case is one whose sentences are exactly equal to the Variant's. That's the case when the Variant does not use other declarations and it explicits all the input test data to use.

For comparison, whether we simply remove with "Bob"from the sentenceWhen I fill <#name> with "Bob" , Concordia Compiler will generate a pseudo-random value to complete the sentence, and it will be like this: When I fill <#name> with "A~!39esljdns so2u1 *ns%".

Concordia Compiler uses a single seed to generate pseudo-random numbers in its algorithms and values. Everytime it runs, a new seed is produced. You can set the seed using the CLI parameter --seed. By setting the seed, the compiler will pick the same paths and produce the same values - which is desirable for reproducing the same behavior in the application under test, but not for discovering new defects.

UI Elements

A UI Element represents a user interface element and can denote its expected behavior through properties.

Although a Widget is easy to declare, it can have some shortcomings:

1) It can make a sentence hard to read, specially when it uses XPath or CSS locators.

2. It can be hard to maintain. When a Widget appears in more than one Scenario and its locator needs to change (e.g. you have changed it in the UI), you need to search and replace all its occurrences in the document. With a UI Element, you just need to change the property locator.

3. It does not offer resources for test generation other than its locator.

A UI Element offers properties that can describe the behavior related to it. These properties are used to infer the test data and test oracles to generate.

Example:

These are the UI Element properties:

• id or locator ⇨ how to locate the widget

• type ⇨ widget type, like textbox, button, etc.

By default, declaring a UI Element without properties is the same as declaring it with:

• id or locator equal to the UI Element name in camelCase (e.g., "Net Price" becomes "netPrice"). The adopted case is configurable.

• type equal to textbox

Another example:

Please see the language reference for more details. Let's proceed with the other declarations for now.

Import

A Import declaration allows you to import declarations from another .featurefile. Example:

Imported declarations are Feature, Constants, Table, Database, and UI Element.

Constants

A Constants block defines one or more constants. Every Constant holds an immutable value that is accessible through the declared name. Example:

Constants are accessed by their name between [ and ]. Example:

Constants are global declarations and cannot have duplicate names. You can use them in UI Element properties and Variant sentences.

Table

A Table declares values in rows and columns to use in UI Element properties. Example:

Note that is used to select values from the declared table, which is denoted between [ and ] (like Constants). Table names are global and share the same namespace as Constant names.

Database

A Database provides a way to indicate a connection to an existing . Example:

is used to select values from a table or view from the declared database, which is denoted between [ and ] . Its table or view names are separated by a dot (.). Database names are global and share the same namespace as Tables and Constants.

Test Events

In order to set the execution environment up, testers can define events that can run database commands or command-line scripts. These events can occur before or after Scenarios or Features:

• Before Each Scenario: executes before every test of each Scenario of the current Feature.

• After Each Scenario: executes after every test of each Scenario of the current Feature.

• Before Feature

Test Events are written like Variants, using Given-When-Then. Since they usually perform actions, it is very likely to useWhen in most sentences. Example:

Commands and database scripts are denoted between apostrophe (').

Line comments

Common line comment

It makes the compiler to ignore a piece of text. Example:

Special line comment

They are:

• #language: <value> defines the document's language, where <value> is an available language such as en (English) or pt (Portuguese).

• #locale: <value> defines document's locale, where

Example:

Tag

A tag adds metadata to a declaration.

A tag starts with @. Multiple tags can be declared in a single line. Examples:

Some tags can receive parameters. Examples:

Special tag

A special tag changes the compiler's behavior. These are special tags:

• @scenario( <number> ): references a by its index, starting at 1.

• @variant( <number> ): references a by its index, starting at 1.

Reserved for future use:

• @global makes a UI Element globally accessible. Global UI Elements must not have the same name. Local UI Elements (the default) overrides global UI Elements.

• @extends( <name> )

Common tag

A common tag is ignored by the compiler and aims to provide meaningful metadata for a declaration. Any tag that is not a special tag is considered a common tag. Examples:

• @critical may indicate that the current feature is critical for the project;

• @issue(20) may indicate that a feature (or scenario or variant) is being implemented in the Issue 20 from the Issue control system.

Feature

Desired behavior of a software system

A Feature can be a piece of functionality or some needed, wished or distinguish characteristic of a system.

Example:

Feature sentences should try to describe the contribution that the Feature brings to the business addressed by the software system. They are optional, but recommended. Example:

In the above example the benefit to the business is to prevent unauthorized access.

Concordia adopts a wide-spread template for describing them: . A User Story template goes like this:

The order of the sentences can vary. A <role> is usually a type of user or stakeholder. A <capability> is what the role wants to perform in order to achieve the <benefit>, which often is the contribution to the business.

State

A certain state of the system

A State is denoted by a text between tilde (~), such as ~a state~, and must be written inside sentences. Example:

A can both produce or require states. A State is produced by a Then sentence, like in the prior example. A State is required by a Given or a When sentence. For instance:

By declaring a required State, you're establishing a dependency among the current and the ones that can produce it. Since that State is often produced by a Variant from another Feature, you have to the corresponding Feature file. Concordia Compiler will only search for states from imported Feature files.

When transforming Variants into , Concordia Compiler will:

• Remove any (Then) sentences with a produced State; and

• Replace any (Given or When) sentences with a required State by sentences from the Variant that can produce it.

Whether there are different s that can produce a (same) required State, Concordia Compiler will be able to combine each of them with the current Variant, aiming to produce that explore different execution paths. For more information, please read the section .

Let's say that we have a simple Login feature like this:

Now let's suppose that a user has to be logged in to access its account details:

Concordia Compiler will produce my-account.testcase with a like the following, that includes sentences from both the features:

Scenario

High-level, business-focused, usage scenario of a Feature.

Example:

Template:

Variant

Test-oriented declaration that describes a possible interaction between a user and the system in order to accomplish a Scenario.

A Variant must use the Given-When-Then template (see ) and first person singular ("I") in its sentences. Concordia Compiler uses techniques to understand them.

Example:

Things that can be part of Variant sentences:

1. References to , such as [New Account Screen]

2. References to , such as {Name} or {New User:Name}.

Best practices

• Make sure that your preconditions are part of the group with Given sentences.

• Make sure that your actions are part of the group with When sentences.

Widgets

Variant sentences can have widgets (of the user interface) denoted between < and >. Example:

Concordia adopts the well-known to locate widgets. Plugins convert these selectors to the format adopted by the testing framework.

Available locators:

Constants

Declaration block with constant values

A Constants block can define one or more constants, preceded by a dash sign (-). Both constant names and values are declared between quotes, except for numeric values - that can be declared without quotes. Example:

Constant values are accessible by their names between [ and ]. Examples for the constants above:

Table

Defines a data table that can be used by UI Elements' properties.

A table must have a unique name. Their rows are denoted between pipe (| ). The first row must provide the column names. Other rows must provide the corresponding data. Example:

Table declarations are transformed into in-memory tables that can be queried by UI Element properties for selecting test data. Tables can be queried using SQL. Like , their names must be referenced between [ and ]. Example:

Database

Declares a database connection that can be used for test data selection

Example 1:

Example 2:

Properties:

UI Element

Import

Imports declarations from a .feature file

Examples:

Test Case

Test Events

Concordia can use databases to retrieve test data or to set the testing environment up before or after running test scripts.

Installation

Installing

Use --db-install plus the driver name. Example:

Available driver names:

Database drivers are installed as development dependencies of your application.

Uninstalling

Use --db-uninstall plus the driver name. Example:

Upgrading

Just uninstall and then install it again.

Usage

A is global declaration. You must declare it once and then import its file in other features. Example:

A database can be referenced inside UI properties and Test Events by its name between [ and ]. Example: [My DB]. To reference a database table, separate the table name from the database name with with a dot (.) - e.g. [My DB.profession]. Let's see an example:

Instead of reinventing the wheel, Concordia uses SQL to query and modify databases and files. Even JSON and INI files can be queried with SQL.

In the example above, the property value from the UI Element "Profession" indicates that its (valid) values are retrieved from the table "profession" that belongs to the database declared as "My DB". The test event Before Each Scenario was declared to connect to the database, clean the table "profession", and then insert new records that the UI Element "Profession" will use.

See also

Concordia Compiler uses plug-ins for:

1. setting the testing environment up;

2. generating and executing test scripts;

3. converting and reporting test results.

Every plug-in can generate test scripts for a different testing framework or programming language.

Plug-ins

Notes:

• (*) RTS means "Requires a ", that is, if it requires a testing server to run the test scripts (e.g., Selenium Standalone).

• (**) Appium requires Java for testing web-based mobile applications or using the Android SDK.

• codeceptjs-testcafe uses the frameworks and , and works with probably any browser.

Installation

Use --plugin-install plus the plug-in name. Example:

Upgrade

Just uninstall the plug-in and then install it again. Example:

How to use a plug-in

Concordia has the following plug-in commands:

• plugin-install to install a plug-in

• plugin-uninstallto uninstall a plug-in

• plugin-serve to start a testing server using the plug-in

All but the latter command (plugin-list) require a plug-in name.

👉 Whether you have a with the property plugin defined, you can omit the plugin name from a command. Example:

Commonly used commands

Starting a testing server

Some plug-ins (WebDriverIO, Appium) require a testing server to execute test scripts. A testing server controls a browser or an emulator during tests.

We recommend to open a new terminal/console and then start the testing server:

The testing server will remain open. To stop it later, type Ctrl + C.

Generating and executing test scripts

Generating test scripts without executing them

You can use --no-run to avoid running test scripts and use --no-result to avoid reading the last report with test results. Test scripts will be generated but not executed.

Executing existing test scripts (without generating them)

You can use --no-script to avoid generating test scripts. Only existing test scripts will be executed.

See also

Let's suppose that we have to create two simple features for a web application: "Login" and "Add Product". After discussing them with stakeholders and the team, we created the following Concordia specification (see the two tabs):

Feature: Login
  As a user
  

import "login.feature"

Feature: Add Product
  As an employee
  I would like to add a product
  In order to manage its data and sale it
  
Scenario: Add a new product with basic data
  Given that I am logged in
  When I add a product with SKU, description, department, price, and quantity
  Then I am able to register it
  
  Variant: Basic data
    Given that I have a ~user logged in~
      and I visit the [Product Screen]
    When I fill {SKU}
      and I fill {Description}
      and I fill {Price}
      and I fill {Quantity}
      and I click on {Save}
    Then I have ~product registered~
      and I see "Saved."

UI Element: SKU
 - required
   Otherwise I see "Please inform the SKU."
 - format is "/[A-Z]{3}\-[0-9]{3}/"
   Otherwise I see "Invalid SKU format. Please use 3 letters, dash, 3 numbers." 
    
UI Element: Description
  - minimum length is 2
    Otherwise I see "Description needs at least 2 characters."
  - maximum length is 100
    Otherwise I see "Description must have at most 100 characters."    
    
UI Element: Price
  - required 
    Otherwise I see "Please inform the price."  
  - minimum value is 0.01
    Otherwise I see "Minimum price is one cent."
    
UI Element: Quantity
  - data type is integer

UI Element: Save
  - type is button    

Constants:
  "Product Screen" is "/products/new"

These features could correspond to the following sketches:

The specification helps stakeholders, analysis, testers and developers to create a shared understanding about the application and its business rules. It helps with the requirements validation, knowledge documentation, project design, implementation and testing.

Different from Use Cases and other requirements specification formats, with Concordia you probably don't need to define different scenarios for validation. Instead of having to manually define a set scenarios for validation, you just need to define how the UI elements should behavior - in their declaration - and the compiler will generate different scenarios for you in the form of Test Cases. Whether your application has a lot of such rules, your team will certainly benefit from it.

<TO-DO (UNDER CONSTRUCTION)>

that a Variant express a possible interaction between a user and the application in order to complete its (business-focused) Scenario. lt follows the syntax and always uses the word "I" to represent the user or user role denoted in the Feature's description.

Let's run Concordia Compiler with a to produce the same results over and over again:

It will generate login.testcase with the following content:

If a seed is not given, Concordia Compiler assumes the current date and time. The seed is always printed in the console so that you or your team can reproduce the same paths, test data, and test oracle that were able to expose a bug in your application. Here we are giving it for a sake of reproduction.

Using the same seed over and over again will make Concordia Compiler produce the same results. Options no-run and no-result avoid running the test scripts and getting execution results.

Execution without generation

This will not generate test cases or test scripts, but it will execute them and get their results:

Output

login.testcase:

test/login.js:

3. Analyze the results

Whether you ran the test scripts above, you probably saw they fail. That's because they didn't find a web application running at http://localhost/login or because the application was found but it did not match the expected behavior. You may adapt your application and run the tests again.

Concordia shows a report that indicates the failures' locations. They help you to decide if a failure was caused by the application under test (e.g., it did not behave as expected) or because of the requirements specification (e.g., it is outdated in relation to the application).

Now keeping your specification updated has a new clear benefit: you can use it to generate tests and discover existing defects in your application!

See also

Imagine a solution in which you can write software requirements using an Agile-style, restricted natural language and it automatically generates for you using effective testing techniques such as , , and and it can also use test data from . Well, that's Concordia. Its solution is composed by a metalanguage and a compiler.

To cite Concordia in a paper, please cite its PhD Thesis:

Concordia Compiler infers test cases, test data and test oracles from specification files written in Concordia and tries to cover a large number of paths that can expose defects in your application. Its approach keeps the specification relevant, up-to-date and more useful for stakeholders, testers, and developers than other documentation formats.

Basic syntax is:

where:

• A parameter between[ and ] means that it is optional.

Information about breaking changes.

From 1.x to 2.x

Please read .

Tools

• Katalon-Concordia is a browser extension for Chrome and Firefox that converts recordings from Katalon Recorder (which is a capture-replay tool - that is, it records all the actions that you perform while interacting with the application under test in order the reproduce them later) to Variant sentences. That's very handful for discovering elements' identification in web applications (e.g., their id properties or their ) and obtaining Variant sentences with the corresponding interaction.

• is a command-line tool that generates User Interface Prototypes (UIP) from Concordia features. It can help discussing features with stakeholders and accelerating their construction. Currently there is a customizable plug-in for HTML-based interfaces.

• is a command-line tool that generates an HTML page with an interactive graph that shows features files' relationships.

Publications

1. PINTO, Thiago Delgado. Unifying Agile requirements specification quality control and implementation conformance assurance. 2018. Doctoral Thesis. Pontifical Catholic University of Rio de Janeiro, Informatics Department, 2018.

2. PINTO, Thiago Delgado; COSTA, Pablo Veiga; GONÇALVES, Willian Inácio. User Interface Prototype Generation from Agile Requirements Specifications written in Concordia. 2019. WebMedia'19: Proceedings on the 25th Brazilian Symposium on Multimedia and the Web, Rio de Janeiro, 2019. ACM. DOI:

⚠ UNDER CONSTRUCTION ⚠

In software testing, test coverage is some kind of metric that helps you to understand what parts of your application (or code) are exercised by your tests. The number of feasible paths through them grows exponentially with an increase in application size and can even be infinite in the case of applications with unbounded loop iterations. That is a problem called path explosion problem. Concordia Compiler deals with it by providing sets of combination and selection strategies, and trying to achieve full path coverage over time.

Test Coverage

• All the Features, Scenarios, and Variants are covered by default.

• The CLI parameter --files can filter the .feature files to be considered.

• The CLI parameter --ignore

State-based Strategies

In Concordia, you can declare a State in a Variant sentence using a text between tile (~), like this:

There are three types of State:

1. Precondition: when declared in a Given sentence;

2. State Call: when declared in a When sentence;

3. Postcondition: when declared in a

Both Preconditions and State Calls are considered required States. That is, they denote a dependency of a certain State of the system that needs to be executed. A Precondition needs to be executed before the Variant starts, and a State Call needs to be executed during the Variant's execution.

A Postcondition is a produced State, that is, a state of the system produced by a successful execution of a Variant. Therefore, whether something goes wrong during a Variant's execution, it will not produce the declared State.

When the current Variant requires a State, Concordia Compiler will look for imported Features' Variants able to produce it. To generate complete Test Cases for the current Variant, it will:

1. Select the Variants to combine;

2. Generate successful test scenarios for the selected Variants;

3. Select the successful test scenarios to combine;

Steps 1 and 3 can adopt different strategies. Concordia Compiler let's you:

• Parameterize how the Variants will be selected, using --comb-variant; and

• Parameterize how the successful test scenarios will be combined, using --comb-state.

Variant selection

Available strategies for --comb-variant:

• random: Selects a random Variant that produces the required State. That's the default behavior;

• first: Selects the first Variant that produces the required State;

• fmi

Example:

State combination