1 of 25

Concordia

Quick Start

Welcome to the Concordia 2 documentation! This page will help you to get started fast. If you run into problems, you can find help on our .

See in case you don't know it yet.

System Requirements

Concordia Compiler requires only version 12.20 (or above) and works on Linux, MacOS and Windows.

In this Quick Start, we'll create a simple specification of an interaction with a search page and use . Therefore, download it before continuing.

Step 0: Initialize

If you are starting a new project or whether your project does not have a file named package.json, then run:

You can use NPM, Yarn or PNPM.

Step 1: Install

Install Concordia Compiler :

It will install version 2 that is in alpha stage, although stable.

Note that the needed package name is concordialang (with lang). After installing it, we'll always use npx concordia to execute it.

Step 2: Configure

Step 3: Create a feature

Create the folderfeatures :

Now create the file search.feature withinfeatures, with the following content:

About the file:

Feature and Scenario are high-level, business-focused descriptions about the problem to solve. Their sentences are not used to generate test cases. The above example does not describe them.
A Variant describes the expected interaction with the application's user interface (UI) in order to perform a Scenario. Thus, a Variant uses technological vocabulary.
In Concordia, all the interactions with the UI use first person singular ("I"). That "I" represents the actor that is interacting with the application (in the example above, a visitor).

Step 4: Execute

Finally, run

It will:

set the testing environment up (once);
generate a test case file and transformed it into a test script file;
execute the test script file; and
report the test script results.

Your browser should open automatically during this process and the console will report the execution results.

Your directory should now look like this:

The file features/search.testcase should look like this:

The Test Case above was produced from the Variant declared in features/search.feature. Some notes about it:

The import clause (line 5) imports the declared file's content.
The tag @generated (line 7) indicates that the Test Case was produced automatically.
The tag @scenario(1) (line 8) indicates that the Test Case belongs to the first Scenario (1).
The tag @variant(1) (line 9) indicates that the Test Case belongs to the first Variant (1) of the Scenario declared previously.
The Test Case (line 10) is named using the Variant's name plus dash (-) and some (incremental) number. Its content is basically the same as the Variant's, since we did not use any other declarations.

The file test/search.js contains the test script produced from features/search.testcase using the selected plug-in. It also contains line comments with references to their corresponding lines and columns in the .testcase file:

Notes on installation options

You can opt to install Concordia Compiler locally (per project) or globally.

Local installation (recommended)

does not require administrative privileges (i.e., using sudo on Linux or MacOS) to install;
allows every project to have its own, independent installation;

Global installation

requires administrative privilegies (i.e., using sudo on Linux or MacOS) to install;
lets you execute the compiler direcly from any folder;
needs more attention when upgrading, especially for plug-ins.

Example:

Note: On Windows, you must omit sudo.

Additional tips:

Whether Windows requires administrative privileges to install, right click Windows (Start) menu, click "Command Prompt (Admin)" or alternatively "Windows PowerShell (Admin)", and then proceed with the installation command.
To upgrade Concordia later, change the installation command from install to upgrade.

Introduction

Understanding Concordia

Imagine a solution in which you can write software requirements using an Agile-style, restricted natural language and it automatically generates for you using effective testing techniques such as , , and and it can also use test data from . Well, that's Concordia. Its solution is composed by a metalanguage and a compiler.

Concordia is also who was the personification of "concord" or "agreement". The idea is that the metalanguage help users, stakeholders, and software teams to discussing and reaching an agreement about the software requirements. A shared understanding about the requirements is essential to create the right software.

Concordia metalanguage

Concordia allows you to focus on writing your application's business rules and defining its expected behavior for when users interact with it. Stakeholders, business analysts, testers, and programmers can now discuss the application using a single language, a single source of truth. Business-focused and testing-focused declarations are clearly separated, so you can discuss technological details only with the interested parties.

With Concordia, you can write both and requirements, although it produces only functional test cases. Because it adopts -style declarations, you don't have to write it so formally (like usually do) and you can use plain text - which is easy to evolve with your codebase.

We invite you to and see how it is easy to understand. It is currently available in English (en) and Portuguese (pt).

Concordia compiler

Concordia compiler mix techniques with and to understand declarations in Concordia Language - which uses restricted natural language. After detecting declarations, it infers test scenarios, test data, and test oracles to generate test cases in a framework-independent format (see the ). Finally, it uses a plug-in created for a specific testing framework to generate test scripts (that is, source code), execute them and read their results. You don't need to write code. And the compiler checks your declarations for logic errors and other possible problems.

Follow the to see it in action! 😉

Concordia plug-ins

Concordia compiler uses plug-ins to transform Abstract Test Cases (ATS) into test scripts - that is, into source code. Every plug-in can be implemented to generate source code for a specific programming language and testing framework - in Java, Python, Go, PHP or whatever the language you prefer. The generated source code is not tied to JavaScript or TypeScript.

See the and for your favorite programming language and testing framework.

A Small Example

This example covers the generation of test data and oracles and the combination of features using states.

Let's suppose that we have to create two simple features for a web application: "Login" and "Add Product". After discussing them with stakeholders and the team, we created the following Concordia specification (see the two tabs):

These features could correspond to the following sketches:

The specification helps stakeholders, analysis, testers and developers to create a shared understanding about the application and its business rules. It helps with the requirements validation, knowledge documentation, project design, implementation and testing.

Different from Use Cases and other requirements specification formats, with Concordia you probably don't need to define different scenarios for validation. Instead of having to manually define a set scenarios for validation, you just need to define how the UI elements should behavior - in their declaration - and the compiler will generate different scenarios for you in the form of Test Cases. Whether your application has a lot of such rules, your team will certainly benefit from it.

<TO-DO (UNDER CONSTRUCTION)>

It will generate login.testcase with the following content:

If a seed is not given, Concordia Compiler assumes the current date and time. The seed is always printed in the console so that you or your team can reproduce the same paths, test data, and test oracle that were able to expose a bug in your application. Here we are giving it for a sake of reproduction.

Using the same seed over and over again will make Concordia Compiler produce the same results. Options no-run and no-result avoid running the test scripts and getting execution results.

Execution without generation

This will not generate test cases or test scripts, but it will execute them and get their results:

Output

login.testcase:

test/login.js:

3. Analyze the results

Whether you ran the test scripts above, you probably saw they fail. That's because they didn't find a web application running at http://localhost/login or because the application was found but it did not match the expected behavior. You may adapt your application and run the tests again.

Concordia shows a report that indicates the failures' locations. They help you to decide if a failure was caused by the application under test (e.g., it did not behave as expected) or because of the requirements specification (e.g., it is outdated in relation to the application).

Now keeping your specification updated has a new clear benefit: you can use it to generate tests and discover existing defects in your application!

Using a Plug-in

Concordia Compiler uses plug-ins for:

setting the testing environment up;
generating and executing test scripts;
converting and reporting test results.

Every plug-in can generate test scripts for a different testing framework or programming language.

Plug-ins

Notes:

(*) RTS means "Requires a ", that is, if it requires a testing server to run the test scripts (e.g., Selenium Standalone).
(**) Appium requires Java for testing web-based mobile applications or using the Android SDK.
codeceptjs-testcafe uses the frameworks and , and works with probably any browser.
codeceptjs-playwright uses the frameworks and ,
requires Node 10.14 or above, and works with Chromium, Firefox, and Safari.
codeceptjs-webdriverio uses the frameworks and , and works with Chrome, Firefox, IE, and Edge.
codeceptjs-appium uses the frameworks and , and requires some to work.
codeceptjs-testcafe and codeceptjs-playwright are only available for Concordia Compiler 2 or above.

Installation

Use --plugin-install plus the plug-in name. Example:

You can also use NPM for installing a plug-in. In this case, you must prefix the plug-in name withconcordialang- . Example: npm install -D concordialang-codeceptjs-testcafe.

Upgrade

Just uninstall the plug-in and then install it again. Example:

How to use a plug-in

Concordia has the following plug-in commands:

plugin-install to install a plug-in
plugin-uninstallto uninstall a plug-in
plugin-serve to start a testing server using the plug-in
plugin to use a plug-in
plugin-info to show information about a plug-in
plugin-list to list installed plug-ins

All but the latter command (plugin-list) require a plug-in name.

Commonly used commands

You can omit the argument <plugin> if you have a configuration file with the property "plugin" defined.

Starting a testing server

Some plug-ins (WebDriverIO, Appium) require a testing server to execute test scripts. A testing server controls a browser or an emulator during tests.

We recommend to open a new terminal/console and then start the testing server:

Naturally, you must replace <plugin> with the plugin name.

The testing server will remain open. To stop it later, type Ctrl + C.

Generating and executing test scripts

Whether your plug-in needs a testing server, start it beforehand.

Generating test scripts without executing them

You can use --no-run to avoid running test scripts and use --no-result to avoid reading the last report with test results. Test scripts will be generated but not executed.

Executing existing test scripts (without generating them)

You can use --no-script to avoid generating test scripts. Only existing test scripts will be executed.

Using Databases

Concordia can use databases to retrieve test data or to set the testing environment up before or after running test scripts.

Installation

Installing

To reduce the number of downloaded packages, Concordia Compiler 2 does not install database drivers by default anymore - as version 1 did.

Use --db-install plus the driver name. Example:

npx concordia --db-install mysql

Available driver names:

Driver Name

Database

note

csv

CSV files

xlsx

Excel files

firebase

Firebase

ini

INI files

json

JSON files

Already installed

mysql

MySQL

adodb

MS Access

Windows only

mssql

MS SQL Server

postgres

PostgreSQL

sqlite

SQLite

Database drivers are installed as development dependencies of your application.

You can also install or uninstall them with NPM, prefixing the driver name with database-js-. Example: npm i -D database-js-mysql.

Uninstalling

Use --db-uninstall plus the driver name. Example:

npx concordia --db-uninstall mysql

Upgrading

Just uninstall and then install it again.

Usage

A Database is global declaration. You must declare it once and then import its file in other features. Example:

Database: My DB
  - type is "mysql"
  - host is "127.0.0.1"
  - name is "acme"
  - username is "tester"
  - password is "secret"
  
Database: Another DB
  - type is "json"
  - path is "./db.json"

A database can be referenced inside UI properties and Test Events by its name between [ and ]. Example: [My DB]. To reference a database table, separate the table name from the database name with with a dot (.) - e.g. [My DB.profession]. Let's see an example:

import "db.feature"

# ... other declarations here ...

UI Element: Profession
  - value comes from "SELECT name FROM [My DB.profession]"
    Otherwise, I see "Profession not found."
  - required
    Otherwise, I see "Please select the profession."
    
Before Each Scenario:
  Given that I connect to [My DB]
  When I run the script 'DELETE FROM profession'
    and I run the script 'INSERT INTO profession (name) VALUES ("Enginner"), ("Accountant")'

Instead of reinventing the wheel, Concordia uses SQL to query and modify databases and files. Even JSON and INI files can be queried with SQL.

In the example above, the property value from the UI Element "Profession" indicates that its (valid) values are retrieved from the table "profession" that belongs to the database declared as "My DB". The test event Before Each Scenario was declared to connect to the database, clean the table "profession", and then insert new records that the UI Element "Profession" will use.

Related things

Tools

is a browser extension for Chrome and Firefox that converts recordings from (which is a - that is, it records all the actions that you perform while interacting with the application under test in order the reproduce them later) to Variant sentences. That's very handful for discovering elements' identification in web applications (e.g., their id properties or their ) and obtaining Variant sentences with the corresponding interaction.
is a command-line tool that generates User Interface Prototypes (UIP) from Concordia features. It can help discussing features with stakeholders and accelerating their construction. Currently there is a customizable plug-in for HTML-based interfaces.
is a command-line tool that generates an HTML page with an interactive graph that shows features files' relationships.

Publications

PINTO, Thiago Delgado. Unifying Agile requirements specification quality control and implementation conformance assurance. 2018. Doctoral Thesis. Pontifical Catholic University of Rio de Janeiro, Informatics Department, 2018.
PINTO, Thiago Delgado; COSTA, Pablo Veiga; GONÇALVES, Willian Inácio. User Interface Prototype Generation from Agile Requirements Specifications written in Concordia. 2019. WebMedia'19: Proceedings on the 25th Brazilian Symposium on Multimedia and the Web, Rio de Janeiro, 2019. ACM. DOI:

Language

Concordia Language

Informal specification

Line comments

Common line comment

It makes the compiler to ignore a piece of text. Example:

Special line comment

Local declaration.
At most one declaration per file.
Take precedence over a CLI option or configuration file option.

They are:

#language: <value> defines the document's language, where <value> is an available language such as en (English) or pt (Portuguese).
#locale: <value> ~~defines document's locale, where~~ <value> ~~is an available local such as~~ en-US ~~(USA English) or~~ pt-BR ~~(Brazilian Portuguese).~~ NOT AVAILABLE YET.

Example:

Feature

Desired behavior of a software system

Global declaration with unique name, at most one declaration per file.
Sentences are optional and are not used for generating test cases.

A Feature can be a piece of functionality or some needed, wished or distinguish characteristic of a system.

Example:

Feature sentences should try to describe the contribution that the Feature brings to the business addressed by the software system. They are optional, but recommended. Example:

In the above example the benefit to the business is to prevent unauthorized access.

The order of the sentences can vary. A <role> is usually a type of user or stakeholder. A <capability> is what the role wants to perform in order to achieve the <benefit>, which often is the contribution to the business.

State

A certain state of the system

Used for generating test cases.

Remove any (Then) sentences with a produced State; and
Replace any (Given or When) sentences with a required State by sentences from the Variant that can produce it.

Let's say that we have a simple Login feature like this:

Now let's suppose that a user has to be logged in to access its account details:

Scenario

High-level, business-focused, usage scenario of a Feature.

Local declaration with a unique name.
One or more declarations per Feature.
Sentences are optional and are not used for generating test cases.

Example:

Template:

Variant

Test-oriented declaration that describes a possible interaction between a user and the system in order to accomplish a Scenario.

Local declaration, one or more per Scenario.
Sentences are used for generating test cases.

Example:

Things that can be part of Variant sentences:

Values such as "bob@example.com"
Numbers such as 123

Best practices

Make sure that your preconditions are part of the group with Given sentences.
Make sure that your actions are part of the group with When sentences.
Make sure that your post-conditions are part of the group with Then sentences.
Whenever possible, use a single Given , a single When and a single Then sentence. Useand for the other sentences.
Write a single action per sentence.
Do not use andin the middle of a sentence (break it, instead).
Use an additional indentation for and sentences.

Widgets

Variant sentences can have widgets (of the user interface) denoted between < and >. Example:

Available locators:

Constants

Declaration block with constant values

Global declaration (can be used by other files).
At most one declaration per file.

A Constants block can define one or more constants, preceded by a dash sign (-). Both constant names and values are declared between quotes, except for numeric values - that can be declared without quotes. Example:

Constant values are accessible by their names between [ and ]. Examples for the constants above:

Table

Defines a data table that can be used by UI Elements' properties.

Global declaration (can be used by other files).
Zero, one or many declarations per file.

A table must have a unique name. Their rows are denoted between pipe (| ). The first row must provide the column names. Other rows must provide the corresponding data. Example:

Database

Declares a database connection that can be used for test data selection

Global declaration.
Names are shared with Constants and Tables.
Allowed more than one declaration per file.
Property values must be declared between quotes (").

Example 1:

Example 2:

Properties:

UI Element

Import

Imports declarations from a .feature file

Local declaration.
Give access to the declarations from the imported file.

Examples:

Test Case

Test Events

Actions

Abstract actions recognized through Natural Language Processing

Concordia Language contains a set of meta-actions that can be used by different testing frameworks for generating test scripts. Concordia Compiler uses Intent Recognition (yes, like chatbots do) to understand sentences and extract the desired action to perform. Later these actions are sent to testing frameworks for conversion into commands.

Concordia actions are recognized in Variants, Test Cases, and Test Events.

The following list presents the available actions and gives some examples on how to use them in sentences. We expect that you do not have to memorize them but, instead, see the things you can do. Try to write the sentences the way you speak, using first person singular (I) . When there is a UI Element in the sentence, embrace it with { and }, like {this}. The compiler will tell you whether it cannot understand some sentence you wrote. If you think it should have understood something, . We'll try to augment the compiler's vocabulary to understand it the next time.

`accept`

Accepts a browser message or app message.

`amOn`

Indicates a webpage or screen in which it is expected to be at.

`append`

Adds a value to an element.

`attachFile`

Attaches a file. It selects a file and confirms its choice (e.g., clicks Open).

`cancel`

Cancels a browser message or app message.

`check`

Checks a checkbox.

`clear`

Empties an input field or browser cookie

`click`

Clicks on something in the screen.

`close`

Closes a tab, a set of tabs, or an app (mobile only).

`connect`

Connects to a database.

`disconnect`

Disconnects from a database.

`doubleClick`

Performs a double click on something.

`drag`

Drags and drops something to a widget.

`fill`

Indicates that a field is being filled. If desired, a value can be given. Otherwise, a value will be generated for the corresponding Test Case.

fill + target

`hide`

Hides something.

`install`

Installs an app.

`maximize`

Maximizes a window or the browser.

`move`

Moves the mouse cursor to a place or element.

`open`

Opens something

`press`

Presses a key or key combination, separated by comma.

Some special keys (case sensitive!):

Add
Alt
ArrowDown or Down arrow
ArrowLeft or Left arrow
ArrowRight or Right arrow
ArrowUp or Up arrow
Backspace
Command
Control
Del
Divide
End
Enter
Equals
Escape
F1 to F12
Home
Insert
Meta
Multiply
Numpad 0 to Numpad 9
Pause
Pagedown or PageDown
Pageup or PageUp
Semicolon
Shift
Space
Subtract
Tab

`pull`

Extracts a device's resource from a path.

`refresh`

Refreshes/reloads the current page.

`remove`

Removes/uninstall an app by its internal name (mobile only).

`resize`

Resizes a window.

`rightClick`

Performs a right click on something.

`run`

Runs a console command or a database script (SQL command).

👉 Commands and SQL scripts must be declared between single quotes ('), in a single line.

👉 SQL scripts must reference their database.

SQL Syntax

JSON and CSV as databases: INSERT accepts JSON objects or arrays as values. Example:

SQL commands are limited to SELECT, UPDATE, INSERT and DELETE. WHERE works well. JOINs are not allowed. GROUP BY is not supported. LIMIT and OFFSET are combined into a single LIMIT syntax: LIMIT [offset,]number

`saveScreenshot`

Takes a screenshot an saves into a file.

`scrollTo`

Scrolls to a certain element.

`see`

Indicates that something can be seen. You can also negate the sentence to indicate something cannot be seen.

`select`

Selects a value for an element.

`shake`

Shakes the device - MOBILE ONLY.

`swipe`

Performs a swipe action - MOBILE ONLY

`switch`

Switches to a certain iframe and back to the application page; OR
Switches to a certain tab; OR
Switches an app to native mode or web mode - MOBILE ONLY.

NOTE: When switching to an iframe, all the commands that follow it will be applied to it.

`tap`

Performs a tap on an element - MOBILE ONLY

`uncheck`

Unchecks a checkbox.

`wait`

Wait for something.

How It Works

Upgrade

How to Upgrade

Versioning

Concordia Compiler's versions are based on .

Although Semantic Versioning is conceived for s instead of for applications, we adopt a very similar convention. Thus, changes become predictable and you can know, from the version numbers, when a version is no more compatible with a previous version.

Given a version MAJOR.MINOR.UPDATE:

MAIOR is increased when the Compiler or the Language is no more compatible with the previous version.
MINOR is increased when adding functionality in a backwards-compatible manner.
UPDATE is increased when there are fixes, little changes or little novelties - all of them backwards-compatible.

Examples:

0.2.0 is compatible with 0.1.0
0.1.1 is compatible with 0.1.0
1.0.0 is not compatible with 0.2.0

Upgrade

NPM upgrades without breaking compatibility (when MINORorUPDATE changes).

For migrating a MAJOR version (e.g., 1.x to 2.x) please read our .

To upgrade a local installation:

To upgrade a global installation:

Development

Actions

Abstract actions recognized through Natural Language Processing

Concordia actions are recognized in Variants, Test Cases, and Test Events.

`accept`

Accepts a browser message or app message.

`amOn`

Indicates a webpage or screen in which it is expected to be at.

`append`

Adds a value to an element.

`attachFile`

Attaches a file. It selects a file and confirms its choice (e.g., clicks Open).

`cancel`

Cancels a browser message or app message.

`check`

Checks a checkbox.

`clear`

Empties an input field or browser cookie

`click`

Clicks on something in the screen.

`close`

Closes a tab, a set of tabs, or an app (mobile only).

`connect`

Connects to a database.

The next sentence is for only:

`disconnect`

Disconnects from a database.

The next sentence is for only:

`doubleClick`

Performs a double click on something.

`drag`

Drags and drops something to a widget.

`fill`

Indicates that a field is being filled. If desired, a value can be given. Otherwise, a value will be generated for the corresponding Test Case.

fill + target

`hide`

Hides something.

`install`

Installs an app.

`maximize`

Maximizes a window or the browser.

`move`

Moves the mouse cursor to a place or element.

`open`

Opens something

`press`

Presses a key or key combination, separated by comma.

Some special keys (case sensitive!):

Add
Alt
ArrowDown or Down arrow
ArrowLeft or Left arrow
ArrowRight or Right arrow
ArrowUp or Up arrow
Backspace
Command
Control
Del
Divide
End
Enter
Equals
Escape
F1 to F12
Home
Insert
Meta
Multiply
Numpad 0 to Numpad 9
Pause
Pagedown or PageDown
Pageup or PageUp
Semicolon
Shift
Space
Subtract
Tab

`pull`

Extracts a device's resource from a path.

`refresh`

Refreshes/reloads the current page.

`remove`

Removes/uninstall an app by its internal name (mobile only).

`resize`

Resizes a window.

`rightClick`

Performs a right click on something.

`run`

Runs a console command or a database script (SQL command).

👉 Available for only.

👉 Commands and SQL scripts must be declared between single quotes ('), in a single line.

👉 SQL scripts must reference their database.

SQL Syntax

Concordia Compiler uses to access databases and files through SQL-like commands. The supported SQL syntax may vary from one database to another. Please see the .

JSON and CSV as databases: INSERT accepts JSON objects or arrays as values. Example:

Excel and Firebase databases: Syntax similar to JSON and CSV databases. However, it has some limitations, as pointed out in :

SQL commands are limited to SELECT, UPDATE, INSERT and DELETE. WHERE works well. JOINs are not allowed. GROUP BY is not supported. LIMIT and OFFSET are combined into a single LIMIT syntax: LIMIT [offset,]number

INI databases: INSERT and DELETE are not supported yet by . UPDATE example:

SQLite databases: Currently uses that doesn't persist the changes in the file (only in memory).

`saveScreenshot`

Takes a screenshot an saves into a file.

`scrollTo`

Scrolls to a certain element.

`see`

Indicates that something can be seen. You can also negate the sentence to indicate something cannot be seen.

`select`

Selects a value for an element.

`shake`

Shakes the device - MOBILE ONLY.

`swipe`

Performs a swipe action - MOBILE ONLY

`switch`

Switches to a certain iframe and back to the application page; OR
Switches to a certain tab; OR
Switches an app to native mode or web mode - MOBILE ONLY.

NOTE: When switching to an iframe, all the commands that follow it will be applied to it.

`tap`

Performs a tap on an element - MOBILE ONLY

`uncheck`

Unchecks a checkbox.

`wait`

Wait for something.

Concordia Language

Informal specification

Line comments

Common line comment

It makes the compiler to ignore a piece of text. Example:

Special line comment

Local declaration.
At most one declaration per file.
Take precedence over a CLI option or configuration file option.

They are:

#language: <value> defines the document's language, where <value> is an available language such as en (English) or pt (Portuguese).
#locale: <value> ~~defines document's locale, where~~ <value> ~~is an available local such as~~ en-US ~~(USA English) or~~ pt-BR ~~(Brazilian Portuguese).~~ NOT AVAILABLE YET.

Example:

Tag

A tag adds metadata to a declaration.

Local declaration.
Allowed more than one tag per language construction.

A tag starts with @. Multiple tags can be declared in a single line. Examples:

...

@critical @slow
Feature: Print Coupon
  
  ...

  @gui
  Variant: Print to PDF
    ...
  
  @device
  Variant: Print to Plotter

Some tags can receive parameters. Examples:

@importance( 6 )
Feature: Register Employee
  ...
  

@extends( Date )
UI Element: Birth Date
  ...

Special tag

A special tag changes the compiler's behavior. These are special tags:

@scenario( <number> ): references a by its index, starting at 1.
@variant( <number> ): references a by its index, starting at 1.
@importance( <number> ): indicates the importance of a declaration. The importance is as high as its number.
@generated: indicates that a was computer-generated.
@fail: indicates that a should fail.
@ignore: whether applied to a , the Variant will not produce Test Cases; whether applied to a it will not produce test scripts.
@generate-only-valid-values: avoids that a 's property be used for generating values considered invalid. This is specially useful for using with masked input fields, in which the system does not let a user to type invalid characters on it. For instance:
```
  UI Element: Year
    - data type is integer
    @generate-only-valid-values
    - format is "/^[0-9]{1-3}$/"
    Otherwise I must see "Year must be a number."
```
The above example will avoid generating invalid input values (e.g., "A") for testing the format of the Year.

Reserved for future use:

@global makes a UI Element globally accessible. Global UI Elements must not have the same name. Local UI Elements (the default) overrides global UI Elements.
@extends( <name> )

Common tag

A common tag is ignored by the compiler and aims to provide meaningful metadata for a declaration. Any tag that is not a special tag is considered a common tag. Examples:

@critical may indicate that the current feature is critical for the project;
@issue(20) may indicate that a feature (or scenario or variant) is being implemented in the Issue 20 from the Issue control system.

Feature

Desired behavior of a software system

Global declaration with unique name, at most one declaration per file.
Sentences are optional and are not used for generating test cases.
Business-oriented sentences, written as a .

A Feature can be a piece of functionality or some needed, wished or distinguish characteristic of a system.

Example:

Feature: Administrator Login

Feature sentences should try to describe the contribution that the Feature brings to the business addressed by the software system. They are optional, but recommended. Example:

Feature: Login
  As a registered user
  I would like to access to the system using my credentials
  So that the system prevents unauthorized access

In the above example the benefit to the business is to prevent unauthorized access.

Concordia adopts a wide-spread template for describing them: . A User Story template goes like this:

As a <role>
I would like to <capability>
So that <benefit>

State

A certain state of the system

Local declaration, only accepted insidesentences.
Used for generating test cases.

A State is denoted by a text between tilde (~), such as ~a state~, and must be written inside sentences. Example:

Then I have a ~user logged in~

A can both produce or require states. A State is produced by a Then sentence, like in the prior example. A State is required by a Given or a When sentence. For instance:

Given that I have ~user logged in~

By declaring a required State, you're establishing a dependency among the current and the ones that can produce it. Since that State is often produced by a Variant from another Feature, you have to the corresponding Feature file. Concordia Compiler will only search for states from imported Feature files.

When transforming Variants into , Concordia Compiler will:

Remove any (Then) sentences with a produced State; and
Replace any (Given or When) sentences with a required State by sentences from the Variant that can produce it.

Whether there are different s that can produce a (same) required State, Concordia Compiler will be able to combine each of them with the current Variant, aiming to produce that explore different execution paths. For more information, please read the section .

Let's say that we have a simple Login feature like this:

Feature: Login

Scenario: Sucessful login

  Variant: Login with username and password
    Given that I am on the [Login Screen]
    When I enter with "bob" in {Username}
      and I enter with "123456" in {Password}
      and I click on {OK}
    Then I see "Welcome"
      and I have ~user logged in~


Constants:
  - "Login Screen" is "/login"
  
UI Element: Username

UI Element: Password

UI Element: OK
  - type is button

Now let's suppose that a user has to be logged in to access its account details:

import "login.feature"

Feature: My Account

  Scenario: See my account data
  
    Variant: Show basic data by default
      Given that I have ~user logged in~
        and I am on the [Account Screen]
      Then I see {User} with "bob"
        and I see {Name} with "Robert Downey Jr"


Constants:
  - "Account Screen" is "/account"
        
UI Element: User

Concordia Compiler will produce my-account.testcase with a like the following, that includes sentences from both the features:

import "my-account.feature"

@scenario(1)
@variant(1)
Test Case: See my account data - 1
  Given that I am on the "/login"
  When I enter with "bob" in <username>
    and I enter with "123456" in <password>
    and I click on <ok>
  Then I see "Welcome"
  Given that I am on the "/account
  Then I see <user> with "bob"
    and I see <name> with "Robert Downey Jr"

Scenario

High-level, business-focused, usage scenario of a Feature.

Local declaration with a unique name.
One or more declarations per Feature.
Sentences are optional and are not used for generating test cases.
Business-oriented sentences that follow the template .

Example:

Feature: Sales Report

  Scenario: Daily sales report
    Given that I am authenticated as a manager
    When I open the sales report
      and I choose the option "Daily report"
    Then I see a list with all the today's sales 
      and the total value

Template:

Given <some context or precondition>
  and <other context or precondition>
  and <other context or precondition>
  ... 
When <some action is carried out>
  and <other action is carried out>
  and <other action is carried out>
  ...
Then <a result or postcondition is observed>
  and <another result or postcondition is observed>
  and <another result or postcondition is observed>
  ...

Variant

Test-oriented declaration that describes a possible interaction between a user and the system in order to accomplish a Scenario.

Local declaration, one or more per Scenario.
Sentences are used for generating test cases.
Test-oriented sentences that follow the template and first-person singular ("I").

A Variant must use the Given-When-Then template (see ) and first person singular ("I") in its sentences. Concordia Compiler uses techniques to understand them.

See the in Variant sentences.

Example:

Feature: New User

Scenario: Successful registration

  Variant: Register with a new account
    Given that I am on the [New Account Screen]
    When  I fill {Name}
      and I fill {Date of Birth}
      and I enter with "bob@example.com" in <#email>
      and I inform 123 in <#pass>    
      and I type 123 in <#confirmation>
      and I click <#ok>
    Then I have a ~registered user~
      and I see "Welcome, Bob" 

...

Things that can be part of Variant sentences:

References to , such as [New Account Screen]
References to , such as {Name} or {New User:Name}.
, such as <#ok>
, such ~registered user~
Values such as "bob@example.com"
Numbers such as 123

Best practices

Make sure that your preconditions are part of the group with Given sentences.
Make sure that your actions are part of the group with When sentences.
Make sure that your post-conditions are part of the group with Then sentences.
Whenever possible, use a single Given , a single When and a single Then sentence. Useand for the other sentences.
Write a single action per sentence.
Do not use andin the middle of a sentence (break it, instead).
Use an additional indentation for and sentences.

Widgets

Variant sentences can have widgets (of the user interface) denoted between < and >. Example:

When I fill <#firstName> with "Alice"

Concordia adopts the well-known to locate widgets. Plugins convert these selectors to the format adopted by the testing framework.

Available locators:

Whether you are testing a web application, you may find useful. It's a browser extension - available for Google Chrome and Mozilla Firefox - that extends Katalon Recorder to save a recorded interaction as Variant sentences. Using it may reduce the time needed to locate the widgets of an existing application.

Constants

Declaration block with constant values

Global declaration (can be used by other files).
At most one declaration per file.
Constants cannot have the same name of a or a .

Constants:
  - "Home" is "/home"
  - "App Name" is "Acme"
  - "Mininum Age" is 18
  - "Pi" is 3.1416

Constant values are accessible by their names between [ and ]. Examples for the constants above:

Variant: Example
  Given that I visit [Home]
    and I see [App Name]
  # ...
  
UI Element: Age
  - minimum value is [Minimum Age]
    Otherwise I see "Invalid age."
    
UI Element: Some Math Value
  - maximum value is [Pi]

Table

Defines a data table that can be used by UI Elements' properties.

Global declaration (can be used by other files).
Zero, one or many declarations per file.
Tables cannot have the same name of a or a .

A table must have a unique name. Their rows are denoted between pipe (| ). The first row must provide the column names. Other rows must provide the corresponding data. Example:

Table: professions
  | name      | min_salary |
  | Magician  | 100000.00  |
  |

Table declarations are transformed into in-memory tables that can be queried by UI Element properties for selecting test data. Tables can be queried using SQL. Like , their names must be referenced between [ and ]. Example:

UI Element: Profession
  - value comes from "SELECT name FROM [professions]"

UI Element: Salary
  - minimum value comes from "SELECT min_salary from [professions] WHERE name = {Profession}"

Database

Declares a database connection that can be used for test data selection

Global declaration.
Names are shared with Constants and Tables.
Allowed more than one declaration per file.
Property values must be declared between quotes (").

Example 1:

Database: My DB
  - type is "mysql"
  - host is "http://127.0.0.1"
  - name is "mydb"
  - username is "admin"
  - password is "p4sss"

Example 2:

Database: Another DB
  - type is "json"
  - path is "C:\path\to\db.json"

Database support needs installation of the corresponding drives. See for more information on how to install them.

Properties:

Property

Description

Required

UI Element

Import

Imports declarations from a .feature file

Local declaration.
Give access to the declarations from the imported file.

Examples:

import "file1.feature"
import "path/to/file2.feature"

Test Case

Test Events

Test Coverage and Techniques

Time is a crucial constraint in software development and software teams often need to focus their test efforts on the most important application paths.

⚠ UNDER CONSTRUCTION ⚠

In software testing, test coverage is some kind of metric that helps you to understand what parts of your application (or code) are exercised by your tests. The number of feasible paths through them grows exponentially with an increase in application size and can even be infinite in the case of applications with unbounded loop iterations. That is a problem called path explosion problem. Concordia Compiler deals with it by providing sets of combination and selection strategies, and trying to achieve full path coverage over time.

Test Coverage

All the Features, Scenarios, and Variants are covered by default.
The CLI parameter --files can filter the .feature files to be considered.
The CLI parameter --ignore can indicate the .feature files to be ignored, when a directory is given.
The tag @ignore can be used to mark a Feature or Variant to be ignored by the test generator. However, it can still be used by other Features or Variants.
~~The~~ ~~tag~~ @importance (~~e.g.~~, @importance( 8 )~~) can be used to denote the importance of a Feature. CLI parameters~~ --sel-min-feature ~~and~~ --sel-max-feature ~~can then be used to filter the features to be considered by the test generator. Example:~~ concordia --sel-min-feature 7 ~~makes the compiler considers the features with importance value of~~ 7 ~~or above. By default, all the features receive an importance value of~~ 5.
Variants are selected and combined using State-based Strategies (see below).
All the UI Elements constraints are covered by default, using a set of Testing Techniques (see below).

State-based Strategies

In Concordia, you can declare a State in a Variant sentence using a text between tile (~), like this:

Given that I have ~payment method selected~

There are three types of State:

Precondition: when declared in a Given sentence;
State Call: when declared in a When sentence;
Postcondition: when declared in a Then sentence.

Both Preconditions and State Calls are considered required States. That is, they denote a dependency of a certain State of the system that needs to be executed. A Precondition needs to be executed before the Variant starts, and a State Call needs to be executed during the Variant's execution.

A Postcondition is a produced State, that is, a state of the system produced by a successful execution of a Variant. Therefore, whether something goes wrong during a Variant's execution, it will not produce the declared State.

When the current Variant requires a State, Concordia Compiler will look for imported Features' Variants able to produce it. To generate complete Test Cases for the current Variant, it will:

Select the Variants to combine;
Generate successful test scenarios for the selected Variants;
Select the successful test scenarios to combine;
Generate (successful and unsuccessful) test scenarios for the current Variant;
Combine the selected successful test scenarios with the test scenarios of the current Variant;
Transform all the test scenarios into test cases (i.e., valued test scenarios).

Steps 1 and 3 can adopt different strategies. Concordia Compiler let's you:

Parameterize how the Variants will be selected, using --comb-variant; and
Parameterize how the successful test scenarios will be combined, using --comb-state.

Variant selection

Available strategies for --comb-variant:

random: Selects a random Variant that produces the required State. That's the default behavior;
first: Selects the first Variant that produces the required State;
fmi: Selects the first most important Variant (since two Variants can have the same importance value) that produces the required State;
all: Selects all the Variants that produce the required State.

Example:

npx concordia --comb-variant=all

State combination

Available strategies for --comb-state:

sre: Single random of each - that is, randomly selects a single, successful test scenario of each selected Variant. That's the default behavior;
sow : Shuffled one-wise - that is, shuffles the successful test scenarios than uses one-wise combination.
ow: One-wise selection;
all: Selects all the successful test scenarios to combine.

Example:

npx concordia --comb-state=all

Full vs random selection

Strategies that use random selection can take different paths every time they are used. Furthermore, they reduce considerably the amount of generated paths - i.e., it avoids "path explosion" - and thus the amount of produced test cases.

Full-selection strategies can be used for increase path coverage. Although, it also increases the needed time to check all the paths, which may be undesirable for frequent tests.

By default, Concordia Compiler uses random selection strategies.

Testing Techniques

Concordia Compiler can infer input test data from Variants, UI Elements, Constants, Tables, and Databases. The more constraints you declare, the more test cases it generates.

Adopted techniques to generate input test data include:

These are well-known, effective black-box testing techniques for discovering relevant defects on applications.

Data Test Cases

We call Data Test Cases those test cases used to generate input test data. They are classified into the following groups: RANGE, LENGTH, FORMAT, SET, REQUIRED, and COMPUTED. The group COMPUTEDis not available on purpose, since a user-defined algorithm to produce test data can have bugs on itself. Thus, one should provide expected input and output values in order to check whether the application is able to correctly compute the output value based on the received input value.

Every group has a set of related data test cases, applied according to the declared constraints and selected algorithms:

Group

Data Test Case

Description

RANGE

LOWEST_VALUE

The lowest value for the data type, e.g., lowest integer

RANDOM_BELOW_MIN_VALUE

A random value below the minimum value

JUST_BELOW_MIN_VALUE

The value just below the minimum value, considering the data type and decimal places if applicable

MIN_VALUE

Exactly the minimum value

JUST_ABOVE_MIN_VALUE

The value just above the minimum value, considering the data type and decimal places if applicable

ZERO_VALUE

Zero (0)

MEDIAN_VALUE

The median between the minimum value and the maximum value

RANDOM_BETWEEN_MIN_MAX_VALUES

A random value between the minimum value and the maximum value

JUST_BELOW_MAX_VALUE

The value just below the maximum value, considering the data type and decimal places if applicable

MAX_VALUE

Exactly the maximum value

JUST_ABOVE_MAX_VALUE

The value just above the maximum value, considering the data type and decimal places if applicable

RANDOM_ABOVE_MAX_VALUE

A random value above the maximum value

GREATEST_VALUE

The greatest value for the data type, e.g., greatest integer

LENGTH

LOWEST_LENGTH

An empty string

RANDOM_BELOW_MIN_LENGTH

A string with random characters and random length, less than the minimum length

JUST_BELOW_MIN_LENGTH

A string with random characters and length exactly below the minimum length

MIN_LENGTH

A string with random characters and length exactly equal to the minimum length

JUST_ABOVE_MIN_LENGTH

A string with random characters and length exactly above the minimum length

MEDIAN_LENGTH

A string with random characters and length equal to the median between the minimum length and the maximum length

RANDOM_BETWEEN_MIN_MAX_LENGTHS

A string with random characters and random length, between the minimum length and the maximum length

JUST_BELOW_MAX_LENGTH

A string with random characters and length exactly below the maximum length

MAX_LENGTH

A string with random characters and length exactly equal to the maximum length

JUST_ABOVE_MAX_LENGTH

A string with random characters and length exactly above the maximum length

RANDOM_ABOVE_MAX_LENGTH

A string with random characters and random length, greater than the maximum length

GREATEST_LENGTH

The greatest length supported for a string (see Notes)

FORMAT

VALID_FORMAT

A value that matches the defined regular expression

INVALID_FORMAT

A value that does not match the defined regular expression

SET

FIRST_ELEMENT

The first element in the defined set or query result

RANDOM_ELEMENT

A random element in the defined set or query result

LAST_ELEMENT

The last element in the defined set or query result

NOT_IN_SET

A value that does not belong to the defined set or query result

REQUIRED

FILLED

A random value

NOT_FILLED

Empty value

~~COMPUTED~~

~~RIGHT_COMPUTATION~~

~~A value generated by the defined algorithm~~

~~WRONG_COMPUTATION~~

~~A value different from that generated by the defined algorithm~~

Maximum length for random string values

By default, the maximum length for randomly-generated string values is 500. This value is used for reducing the time to run test scripts, since long strings take time to be entered.

You can set maximum length using the CLI parameter --random-max-string-size. Example:

npx concordia --random-max-string-size=300

You can also set it in the configuration file (.concordiarc) by adding the property randomMaxStringSize. Example:

"randomMaxStringSize": 300

Properties vs Data Test Cases

Data Test Cases (DTC) are selected for every declared UI Element and its properties. The more properties you declare for a UI Element, the more data you provide for Concordia Compiler to generate DTC.

Example of some evalutations:

When no properties are declared, FILLED and NOT_FILLED are both applied and considered as valid values;
When the property required is declared, NOT_FILLED (empty) is considered as an invalid value;
When the property value is declared:
- if it comes from a set of values (inclusing a query result), all the DTC of the group SET are applied;
- otherwise, ...
When the property minimum value is declared:

There is more logic involved for generating these values. ...

Example 1

Let's describe a user interface element named Salary :

UI Element: Salary
  - data type is double

When no property is defined or only the property data typeis defined,

We defined the property data type as double, since the default data type is string.

Since few restrictions were made, Salary will be tested with the test cases of the group REQUIRED:

FILLED: a pseudo-random double value is generated;
NOT_FILLED: an empty value will be used.

Now let's add a minimum value restriction.

UI Element: Salary
  - data type is double
  - minimum value is 1000.00
    Otherwise I must see "Salary must be greater than or equal to 1000"

Some tests of the group RANGE are now applicable:

LOWEST_VALUE: the lowest possible double is used
RANDOM_BELOW_MIN_VALUE: a random double before the minimum value is generated
JUST_BELOW_MIN_VALUE: a double just below the minimum value is used (e.g., 999.99)
MIN_VALUE: the minimum value is used
JUST_ABOVE_MIN_VALUE: a double just above the minimum value is used (e.g., 1000.01)
ZERO_VALUE: zero (0) is used

Since 1000.00 is the minimum value, the data produced by the tests 1, 2, 3, and 6 of the group VALUE are considered invalid, while 4 and 5 are not. For these tests considered invalid, the behavior defined in Otherwise, that is

    Otherwise I must see "Salary must be greater than or equal to 1000"

is expected to happen. In other words, this behavior serves as test oracle and must occur only when the produced value is invalid.

Unlike this example, when the expected system behavior for invalid values is not specified and a test data is considered invalid, Concordia expects that test should fail. In this case, it generates the Test Case with the tag @fail.

Now let's add maximum value restriction:

UI Element: Salary
  - data type is double
  - minimum value is 1000.00
    Otherwise I must see "Salary must be greater than or equal to 1000"
  - maximum value is 30000.00
    Otherwise I must see "Salary must be less than or equal to 30000"

All the tests of the group RANGE are now applicable. That is, the following tests will be included:

MEDIAN_VALUE: the median between the minimum and the maximum values
RANDOM_BETWEEN_MIN_MAX_VALUES: a pseudo-random double value between the minimum and the maximum values
JUST_BELOW_MAX_VALUE: the value just below the maximum value
MAX_VALUE: the maximum value
JUST_ABOVE_MAX_VALUE: the value just above the maximum value
RANDOM_ABOVE_MAX_VALUE: a pseudo-random double above the maximum value
GREATEST_VALUE: the greatest possible double

The tests from 5 to 7 will produce values considered invalid.

Example 2

Let's define a user interface element named Profession and a table named Professions from which the values come from:

UI Element: Profession
  - type is select
  - value comes from "SELECT name from [Professions]"
  - required

Table: Professions
  | name       |
  | Lawyer     |
  | Accountant |
  | Dentist    |
  | Professor  |
  | Mechanic   |

Applicable test are:

FILLED
NOT_FILLED
FIRST_ELEMENT
RANDOM_ELEMENT
LAST_ELEMENT
NOT_IN_SET

The first two tests are in the group REQUIRED. Since we declared Profession as having a required value, the test FILLED is considered valid but NOT_FILLED is not. Therefore, it is important to remember declaring required inputs accordingly.

The last four tests are in the group SET. Only the last one, NOT_IN_SET, will produce a value considered invalid.

Example 3

In this example, let's adjust the past two examples to make Salary rules dynamic and change according to the Profession.

Firstly, we add two columns the the Professions table:

Table: professions
  | name       | min_salary | max_salary |
  | Lawyer     | 100000     |  900000    |
  | Accountant |  90000     |  800000    |
  | Dentist    | 150000     |  900000    |
  | Professor  |  80000     |  500000    |
  | Mechanic   |  50000     |  120000    |

Then, we change the rules to retrieve the values from the table:

UI Element: Salary
  - data type is double
  - minimum value comes from the query "SELECT min_salary FROM [Professions] WHERE name = {Profession}"
    Otherwise I must see "The given Salary is less than the minimum value"
  - maximum value comes from the query "SELECT max_salary FROM [Professions] WHERE name = {Profession}"
    Otherwise I must see "The given Salary is greater than the maximum value"

The reference to the UI Element {Profession} inside the query, makes the rules of Salary depend on Profession. Every time a Profession is selected, the minimum value and the maximum value of Salary changes according to the columns min_salary and max_salary of the table Professions.

Command Line Interface

CLI

Basic syntax is:

concordia [dir] [options]

where:

A parameter between[ and ] means that it is optional.
[dir] is the directory that contains your .feature files. Whether it is not informed, the tool searches all the subdirectories from the current path.
[options] changes how the tool behaves.

You can run concordia --help to see all the command line interface (CLI) options.

You need to use npx before the commands if you opted for a local installation.

Configuration

--config

Indicates a configuration file

Alias: -c

concordia --config /path/to/.concordiarc

By default the configuration file is loaded from the current directory.

--init

Creates a configuration file and optionally installs plug-ins and databases.

concordia --init

This command asks about your preferences. By hitting Enter without giving an answer, it will select the suggested (default) value. You can use Up and Down to navigate the answers and hit Enter to select it. For multiple selection (when available), hitSpace.

By using a configuration file (e.g. .concordiarc) you avoid informing the configured parameters in the console/terminal. For instance, whether plugin is defined in the configuration file, you don't need to inform --plugin anymore.

CLI options take precedence over values from configuration files.

--save-config

Add CLI parameters to the configuration file

The following example will consider the given directory and add it to the configuration file:

concordia --save-config --directory ./features

Directories and files

--directory

Indicates the directory that has .feature and .testcase files.

Alias: -d

When not given, it will search the current directory and all subdirectories recursively.

We strongly recommend to use it or set it in the configuration file.

Example:

concordia --directory ./features

This is equivalent to:

concordia ./features

You can use it with --no-recursive to disable recursive search.

--file

Specifies .feature files to consider

Aliases: -f, --files

Files must be separated by comma.

concordia --file "file1.feature,/path/to/file2.feature"

Whether the command also informs a directory (e.g. --directory), it searches the files inside the directory. Example:

concordia ./feature --file "f1.feature,f2.feature"

The example above will search for ./feature/f1.feature and ./feature/f1.feature.

--ignore

Specify .feature files to ignore when a directory is informed

Alias: -i

Files must be separated by comma.

concordia ./feature --ignore "f3.feature,f4.feature"

--no-recursive

Disables recursive search when a directory is informed

concordia ./features --no-recursive

Output directories

The directories will be used by a plug-in. We strongly recommend that you specify them in a configuration file.

--dir-result

Specifies the directory for test reports and screenshots.

Alias: -O

concordia --dir-result output

--dir-script

Specifies the directory for test script files.

Alias: -o

concordia --dir-script test

Language and locale

--language

Sets the default language to consider.

Alias: -l

The language code must have only 2 characters and needs to be one of the available languages.

concordia --language pt

--language-list

Lists the available language codes.

concordia --language-list

--locale-list

Lists the available locales.

concordia --locale-list

Processing and output

--just-script

Just generates test scripts

concordia --just-script

--just-spec

Just verifies specification files

concordia --just-spec

--just-test-case

Just generates test cases

concordia --just-script

--just-run

Just execute test script files

concordia --just-run

--no-script

Avoids generating test scripts

concordia --no-script

--no-spec

Avoids processing specification files

concordia --no-spec

--no-test-case

Avoids generating test cases

concordia --no-test-case

--no-result

Avoids reading test scripts' results

concordia --no-result

--no-run

Avoids running test scripts

concordia --no-run

Use -x to combine --no-run and --no-result

concordia -x

--verbose

Uses a verbose output

concordia --verbose

Test script execution

These parameters require a plugin (--plugin) and some plug-ins may not support them.

--instances

Number of parallel instances to execute.

Alias: -I

concordia --instances 2

--headless

Executes browsers in headless mode.

Alias: -H

concordia --headless

--script-file

Sets one or more test script files to execute

Aliases: -F, --script-files

Files must be separated by comma.

concordia --script-file "test1.js,test2.js"

--script-grep

Sets a text or regular expression to filter the files to execute

Alias: -G

concordia --script-grep "Sales"

--target

Sets target browsers or platforms to execute

Alias: -T

Browser or platforms must be separated by comma.

concordia --target "chrome,firefox,edge"

Plug-in

You can omit the prefix concordialang- from plug-in names. For instance, concordialang-codeceptjs-playwright becomescodeceptjs-playwright.

--plugin

Uses the given plug-in (if installed)

Alias: -p

concordia --plugin <plugin>

You can use it with -x to avoid running test scripts and reading their output:

concordia -p <plugin> -x

--plugin-about

Shows information about an installed plug-in

Alias: --plugin-info

concordia --plugin-info <plugin>

--plugin-install

Installs a plug-in

concordia --plugin-install <plugin>

--plugin-list

Lists installed plug-ins

concordia --plugin-list

--plugin-serve

Starts a testing server for a plug-in

Alias: -S

concordia --plugin-serve <plugin>

The testing server will keep running. Thus, it's a good idea to open a separate terminal/console for it. To stop it later, type Ctrl + C.

--plugin-uninstall

Uninstalls a plug-in

concordia --plugin-uninstall <plugin>

Database

See Using Databases for the list of available databases.

--db-install

Installs a database driver

concordia --db-install <database>

--db-list

Lists installed databases

concordia --db-list

--db-uninstall

Uninstalls a database driver

concordia --db-uninstall <database>

Random generation

--random-max-string-size

Maximum random string size to generate, when free values are allowed.

The default value is 500 .

concordia --random-max-string-size 100

--random-min-string-size

Minimum random string size to generate, when free values are allowed.

The default value is 0 .

concordia --random-min-string-size 1

--random-tries

Random tries to generate invalid values.

When there is a set of valid values and any value different from those in the set is considered an invalid, random generation is used to produce an invalid value. This parameter defines how many tries to take. Usually it gets on the first try.

The default value is 5 .

concordia --random-tries 10

--seed

Sets a random seed

concordia --seed="example"

Concordia Compiler can use random selection in its algorithms. By default, it generates the seed using the current date and time. Using the same seed makes the algorithms produce the same output (e.g., same values and paths). While that's interesting for reproducing a past behavior, we don't recommend it for finding new defects. Using new seeds, Concordia Compiler will select different test data, test oracles, and test paths over time.

You can also set the algorithms to use, in order to avoid random selection. Full combination (therefore full coverage of the requirements) can be achieved and it is recommend for complete checking before a new release. However, it may take some time to execute.

Combination strategies

--comb-data

--comb-invalid

--comb-state

--comb-variant

File formats and extensions

You probably don't need to use them, unless you're facing problems to read files.

--line-breaker

Sets the character(s) used for breaking lines.

Since by default the line breaker is detected automatically, this command may not be useful for you in most cases.

concordia --line-breaker "\n"

Note on line breakers:

Linux uses "\n"
MacOS uses "\r"
Windows uses "\r\n"

--encoding

Sets the expected file encoding

Alias: -e

The default is utf8. Possible values are:

ascii
latin1
ucs2 or ucs-2
utf16le or utf-16le
utf8 or utf-8

concordia --encoding latin1

Information

--about

Shows information about the compiler.

concordia --about

--help

Shows the available commands

concordia --help

--newer

Verify if there is a newer version.

concordia --newer

--version

Shows the current version.

concordia --version

Creating a Plug-in

Introduction

Concordia Compiler uses plug-ins for generating test scripts, running them and reading their results.

A plug-in can...

be coded in JavaScript or any language that "transpiles" to JavaScript, such as TypeScript (recommended), Dart, or CoffeeScript.
generate code (test scripts) for any programming language or testing framework able to perform functional tests for web, desktop or mobile applications.

Requirements for a plug-in

Its name must start with concordialang-, for example: concordialang-my-awesome-plugin.
It must be installable with NPM. We recommend you to publish it at NPM too.
It must implement the interface Plugin from concordialang-plugin.
Its package.json file must contain the property concordiaPlugin, like in the following example:

{
   ...
   
   "concordiaPlugin": {
      "file": "dist/index.js",
      "class": "MyAwesomePlugin",
      "serve": "npx command-to-start-my-testing-server"
   }
}

Properties of concordiaPlugin:

file: string: Relative path of the file that contains an implementation of the Plugin interface.
class: string: Class (name) that implementsPlugin.
serve: string: Multi-platform console command to execute in order to start the testing server. Whether the testing framework does not need running a testing server, you can set something like echo \"Running a testing server is not needed.\".

Tasks to perform

A plug-in must deal with three tasks:

Transforming Abstract Test Scripts into test scripts (i.e., source code).
Executing the produced test scripts, regarding some execution options.
Transforming test execution results (produced by the testing framework) to the expected resulting format.

Let's get into the details on how to accomplish them.

Transforming Abstract Test Scripts into test scripts

Think of an Abstract Test Script (ATS) as an object to transform into source code. Basically, it contains a structure that represents the actions to be converted into commands of your preferred programming language.

For instance, consider the following action:

{
    "action": "click",
    "targets": ["#ok"]
}

That could be converted to the following command in Codeception, for PHP:

I->click("#ok");

Or to the following command in Cypress, for JavaScript:

cy.get('#ok').click();

Let's consider a more realistic example, starting with a Test Case in the hypothetical file feature1.testcase:

import "feature1.feature"

@generated
@scenario(1)
@variant(1)
Test Case: Scenario 1 | V1
  Given that I am on "https://myapp.com/register"
  When I fill <#email> with "john.doe@example.com"
    and I fill <#nome> with "John Doe"
    and I click on <#register>
  Then I see "Welcome, John Doe"

That would probably produce an Abstract Test Script like this:

{
    "feature": { "name": "Feature 1" },
    "scenario": { "name": "Scenario 1" },
    "testcases": [
        {
            "scenario": "V1",
            "commands": [
                {
                    "action": "amOn",
                    "values": [ "https://myapp.com/register" ],
                    "location": { "line": 7, "column": 2 }
                },
                {
                    "action": "type",
                    "targets": [ "#email" ],
                    "values": [ "john.doe@example.com" ],
                    "location": { "line": 8, "column": 2 } 
                },            
                {
                    "action": "type",
                    "targets": [ "#name" ],
                    "values": [ "John Doe" ],
                    "location": { "line": 9, "column": 4 }                  
                },                
                {
                    "action": "click",
                    "targets": [ "#register" ],
                    "location": { "line": 10, "column": 4 } 
                },
                {
                    "action": "see",
                    "values": [ "Welcome, John Doe" ],
                    "location": { "line": 11, "column": 2 }
                }                                         
            ]
        }
    ]
}

A plug-in for Cypress could generate the following code from the above object:

context("Feature 1", function() {

    it("Scenario 1 | V1", function() {
        cy.visit("https://myapp.com/register"); // (7,2)
        cy.get("#email").type("john.doe@example.com"); // (8,2)
        cy.get("#name").type("John Doe"); // (9,4)
        cy.get("#register").click(); // (10,4)
        cy.get("body").should("contain", "Welcome, John Doe"); // (11,2)
    });

});

To help generating such commands, you can use some template library such as Mustache (recommended) or Handlebars.

You can observe that after every command there is a line comment containing the corresponding location in the .testcase file. These comments will help to track the specification in case of test script failures.

Executing the test scripts

To execute the produced test scripts, you can use the target framework API or run the tool. Your call.

For example, whether you chose to run Cypress through npx and generate a report using Mocha JUnit Reporter, the command to run could be the following:

npx cypress run --reporter junit

Transforming execution results

Now the plug-in needs to read the report and transform it into an object of the class TestScriptExecutionResult that will be returned to Concordia Compiler.

For instance, the core of the plug-in for CodeceptJS reads the report from a JSON file generated by using CodeceptJS with Mocha Multi Reporters. The current implementation is available in the class ReportConverter.

A plug-in must be able to track back the location of failures or errors in the corresponding .testcase file from a test script. When a test script has a failure or an error, the report saves the stack trace where it occurred. For instance, tests/feature1.js:20:4 indicates, respectively, the file, line and column where the problem happened. You can use these data to read the line of the test script file and retrieve the line comment that contains the corresponding line in the .testcase file.

You can use a regular expression to extract the data from the stack trace, like ReportConverter did in the method extractScriptLocationFromStackTrace. Then you can use the class FileInstrumentationReader from concordialang-plugin to retrieve the location from the line comment. Example:

const scriptLocation = /* extract from stack trace */;

// Suppose that `failureLocation` is:
// {
//    path: "tests/feature1.js",
//    line: 7,
//    column: 8
// }

// Retrieve the comment with the reader:
const specLocation = reader.retrieveSpecLocation(
    scriptLocation
    );
    
// `specLocation` would be:
// {
//    path: "features/feature1.testcase",
//    line: 10,
//    column: 4
// }

You can read these locations in the examples described earlier. Whether tests/feature1.js:7:8 is...

cy.get("#register").click(); // (10,4)

Retrieving the line 10 and column 4 fromfeature1.testcase would give us:

  and I click on <#register>

In TestScriptExecutionResult, both test failures and errors are represented as an exception which includes the properties scriptLocation and specLocation.

Basic guide

We recommend you to follow these steps when creating a plug-in:

1. Open an issue to indicate that you are interested in developing a new plug-in for a certain testing framework or programming language. That is good way to make others aware of your work.

2. Choose a good name for your plug-in and check whether it registered at NPM. Remember that it must start with concordialang-. We recommend that it includes the target framework, tools or language. Example: concordialang-selenium-python to create a plug-in that generates test scripts for the Selenium framework and the Python language. You may use the opened issue (step 1) to get feedback about the name, if you wish.

3. Create a repository for it (e.g., at GitHub, GitLab, BitBucket) and clone the repository.

4. Create a NPM package with npm --init.

5. Add the property concordiaPlugin to your package.json file.

6. Install concordialang-plugin as a dependency:

npm install --save concordialang-plugin

7. Add your files. We recommend the following structure, but you can adapt it as you wish. For example, if you use TypeScript, you will probably add a tsconfig.json file. If you use Jest, your test folder will probably be named __tests__. We also recommend you to add a .editorconfig file and a .gitignore file (the latter should include the folder node_modules).

 ┃
 ┣ src/            🡐 your source code
 ┣ test/           🡐 your tests
 ┣ .editorconfig
 ┣ .gitignore
 ┗ package.json

8. Implement the interface Plugin. We recommend you to take a look at other plugins that did a similar job and encourage you to unit test all your code.

9. Test it with the Concordia Compiler before publishing it. Create a simple project that uses your plug-in. You can use NPM to install your plug-in from a directory in your computer or your remote repository. For example:

cd path/to/your/test-project/
npm install /path/to/your-plugin

10. Publish your plug-in at NPM after making sure that you are using Semantic Versioning. Congratulations! Tell everybody and ask for feedback! ✌

npm publish

What's Next

Ideas and proposals

Work in progress and issues:

Kanban Board (GitHub)
Issues (GitHub)

Logo

Yes, this project does not have a logo yet 😱. Designers interested in helping us (for free 🙏), please contact us on our Slack Channel or open an Issue.

The logo should reflect the idea behind the language:

Concordia is the name of a roman goddess who was the personification of "concord" or "agreement". The idea is that the metalanguage may help users, stakeholders, and the software team to discuss and to reach an agreement about the software requirements. This shared understanding is essential to the software construction.

The tool generates test cases and test scripts that verify if the application under test meets the specified functional requirements. Therefore, the idea of "verification" or "testing" could be in the logo.

Plug-ins

Interested? See how to create a plug-in.

👉 Our next goal is a plugin for Cypress.

Suggestions:

JavaScript and friends: Cypress (web), MacacaJS (web, mobile, desktop)
PHP: Codeception (web)
Java: Selenium (web), Robotium (android), AssertJ Swing (desktop)
Ruby: Capybara (web), Calabash (mobile applications developed with Xamarin)
Python: Selenium (web)
C++: QtTest (desktop), DUnit (desktop)
Delphi: DUnitX (desktop), DUnit (desktop)
Other programming languages or frameworks are welcome! 💖

Language

Variant Background

This language construction is part of Concordia Language since the first version, but it was not implemented yet.

A Variant Background defines steps that must be executed before all the Variants of a certain Scenario. Example:

Feature: Search

  Scenario: Simple Search
  
    Variant Background:
      Given that I visit the [Search Page]
      When I fill {Search} with "Hello"
      
    Variant: Search by clicking on Go
      When I click on {Go}
      Then I see "Hello World!"
      
    Variant: Search by hitting Enter
      When I press "Enter"
      Then I see "Hello World!"

MOTIVATION: It allows to keep initial, repeated steps in a common place.

Constant names without quotation marks

Like this:

Constants:
  - Net Price is 10.0
  - hello is "hello"

MOTIVATION: Simpler syntax.

Visual comparison

Please see Issue #27.

MOTIVATION: It allows to perform visual comparison and detect related bugs.

File content comparison

To use the characters """ (three quotation marks) to denote a text content to be compared. Example:

  ...
  Then I see the file "out.txt" with
"""
This is some text to be compared.
  It will probably
  be compared to a
  text file.
"""

MOTIVATION: It facilitates comparisons with text files or multiple-line strings.

Table matching

To allow a given UI Element or UI Literal to match a certain Table.

Example 1:

Then I see the table {MyTable} as [Some Table]

Example 2:

Then I see the table <myTable> as [Some Table]

In which Some Table is declared like this:

Table: Some Table
| Name  | Age |
| Bob   | 21  |
| Suzan | 25  |

It should make target table's rows to match the declared ones.

MOTIVATION: It facilitates the verification of expected data in tables from a user interface.

NOTES: Probably it requires to convert the Concordia Action to many abstract commands that verify every table row independently. Usually testing frameworks do not provide such kind of verification (for many rows at once).

Date and time expressions inside tables

Use Concordia 2 data and time expressions inside table rows. They must be written between ` or some other character. Example:

Table: Some Table
| Name | Age            |
| Bob  | `18 years ago` |

MOTIVATION: Concordia Language allows date and time expressions inside UI Properties. Using them inside tables can be useful.

Dynamic states

States that vary according to some generated value. Example:

...
    Given that I fill {User}
    and I fill {Pass}
    and I click  on {OK}
  Then I see "Welcome"
    and I have ~{User Type} is logged in~

UI Element: User
  - value comes from "SELECT user FROM [Users]"

UI Element: Pass
  - value comes from "SELECT pass FROM [Users] WHERE user = {User}"

UI Element: User Type
  - value comes from "SELECT user_type FROM [Users] WHERE user = {User}"

Table: Users
  | user  | pass    | user_type |
  | bob   | 123456  | admin     |
  | joe   | 654321  | guest     |
  | alice | 123456  | admin     |

According to the selected user, it will produce a different State. For example, when "bob" is selected, the produced state will be admin is logged in and when "joe" is selected, the produced state will be guest is logged in .

Thus, Features could depend on static or dynamic states.

MOTIVATION: Making states more flexible.

Alternative states

To provide a Given sentence that requires one - and only one - of the provided states. Example:

Variant: Example 1
  Given that I have either ~x~ or ~y~
 ...
    
Variant: Example 2
  Given one of ~x~, ~y~
  ...

MOTIVATION: It allows to perform a XOR operation with the given states using natural language, aiming at choosing one and only one of the given states.

Annotations to parameterize combination strategies

To provide annotations parameterize a test combination strategy for a specific Feature.

For instance, to configure the combination of invalid values for a specific Feature:

@comb-invalid(1)
Feature: Example

The annotations should correspond to the CLI parameters.

MOTIVATION: Increase the flexibility of the test generation procedure.

Multi-line queries

Currently:

- value comes from "SELECT name FROM [MyDB].profession"

Proposal (to accept as valid):

- value comes from "SELECT name
                   FROM [MyDB].profession"`

Alternative proposal:

- value comes from
"""
SELECT name
FROM [MyDB].profession
"""

MOTIVATION: Make it easier to write/read SQL statements.

Compiler

Selection Options

Select a Feature for Test Case generation, without having to include its dependencies:
- by importance value
Select a Scenario for Test Case generation:
- by importance value (tag @importance)
Test Case for test script generation;
- by importance value (tag @importance)

Watch Mode

Provide the option --watch to generate .testcase files when their corresponding .feature file changes or an imported .feature file changes. A new seed must be produced, except when explicitly provided (using --seed or configuration file).

Language Support

Concordia has language constructions that the Compiler does not support yet. Examples:

Variant Background: Implement Variant Background, which is part of the Concordia Language but was not implemented yet by the Concordia Compiler.
Support the tag @ignore in Features and Scenarios. Currently it is supported inVariants and Test Cases. Whether added to a Feature or a Scenario, it would not generate Test Cases.
Consider global UI Elements
Make the tool processing UI Elements tagged with @global.
Allow inheritance of UI Elements
Use @extends( <name> ) to extend another UI Element.
Example:
```
UI Element: Name
- min length is 2
- max length is 100

@extends( Name )
UI Element: Emergency Contact Name

# Emergency Contact Name inherits the properties from Name
```
Multiple declared Tables per query: Allow a query to reference more than one table declared in the specification.
Multiple declared Databases per query: Allow a query to reference more than one database declared in the specification.

New Test Cases

Test cases that explore SQL Injection: Using declared Databases and Queries to formulate Test Cases that generate strings with SQL Injection commands.
Test cases that use naughty strings as test data: Using a list of naughty strings as test data.

UI and Report

It is already possible to generate HTML or even PDF reports using the available plug-ins. However, these reports indicate the frameworks' results, not necessarily the ones from Concordia.

To provide the CLI parameter --report for indicating the corresponding plug-in. Example:

npx concordia --report html

which should be equivalent to

npx concordia --report concordialang-report-html

Tool integration

Integration with text editors

Create new projects for auto-completion plug-ins for text editors such as VS Code, Sublime Text, Atom, etc. Example for VS Code: gherkin-autocomplete.
Be able to run test scripts using Concordia from the IDE.

Integration with reporting tools

Create integration with Allure. Other reporters (e.g., Macaca Reporter) can be added further.

Performance improvements

Avoid generating test scripts when their Test Case files did not change

Keep some hash control or use Git information when available. Hashes can be stored in a .json or .yml file. Example:

{
  "hashes": {
    "feature1.testcase": "ox10JBprHtu5c8822XooloNKUfk=",
    "subdir/feature2.testcase": "DMcj5b67Albe4KhpzyvphC5nVDHn1oCO",
  }
}

Generate unrelated files in parallel

NodeJS has made considerable progress since version 10.5 adding support to Worker Threads. Stable support for Worker Threads was added in version 12 LTS. Although, it would increase minimum requirements to install Concordia Compiler - currently it requires NodeJS 8.0.

Configuration file

Concordia Compiler can use a configuration file in JSON format named .concordiarc. You can generate it by running the following command:

npx concordia --init

You'll be asked about the desired configuration and then the file will be saved.

Example:

{
    "language": "pt",
    "directory": "docs/features",
    "plugin": "codeceptjs-testcafe",
    "dirScript": "test/e2e",
    "dirResult": "test/e2e/output"
}

Configuration file's properties are similar to CLI parameters. The example above defines: "pt" (meaning "Portuguese") as the default language; the directory "docs/features" for features and test cases; the plug-in "codeceptjs-testcafe" for generating test scripts; the directory "test/e2e" for test scripts; and the directory "test/e2e/output" for output files such as report files and screenshots.

CLI options take precedence over values from the configuration file

`directory`

Type

Default value

CLI option

string

"." (current dir)

--directory

Directory to search for .feature and .testcase files.

Example:

{
  "directory": "./features"
}

`recursive`

Type

Default value

CLI option

boolean

true

--recursive

Use recursive directory search.

Example:

{
  "recursive": false
}

`files`

Type

Default value

CLI option

string or string array

(none)

--files

Files to consider, instead of considering directory. The files must be separated by colon.

Example:

{
  "files": "/path/to/file1.feature,/other/file2.feature"
}

Another example:

{
  "files": [
    "/path/to/file1.feature",
    "/other/file2.feature"
  ]
}

We recommend you to configure the property directory whether your project can have new files.

`ignore`

Type

Default value

CLI option

string or string array

(none)

--ignore

Files to ignore, considering the parameter directory. The files must be separated by colon. Example:

{
  "ignore": [
    "/path/to/file1.feature",
    "/other/file2.feature"
  ]
}

`extensions`

To-Do

`encoding`

Type

Default value

CLI option

string

"utf8"

--encoding

File encoding of the files. Accepted values are:

utf8 or utf-8
latin1
ascii
ucs2 or ucs-2
utf16le or utf-16le

Naturally, the files must be edited with the given file encoding.

To-Do

Overview of the Language

Overview of the Concordia declarations:

Features and Scenarios

A Featuredefines a desired behavior for the software, a piece of functionality that gives some business value. A Scenario is a high-level usage scenario of a Feature. A Feature can have many Scenarios.

Since Feature and Scenario are business-oriented declarations, they are not used by Concordia Compiler to generate test cases. You can write them freely. Although, we recommend you to use the User Story template for a Feature and the Given-When-Then template for a Scenario. Example:

Feature: Sales Report
  As a manager
  I would like to generate a sales report
  In order to track company sales

Scenario: Daily sales report
  Given that I am authenticated as a manager
  When I open the sales report
    and I choose the option "Daily report"
  Then I see a list with all the today's sales 
    and the total value

This is the User Story template:

As a <role>
I would like to <goal to perform>
In order to <received benefit> (or So that <received benefit>)

And this is the Given-When-Then (GWT) template:

Given <some context or precondition>
When <some action is carried out>
Then <a result or postcondition is observed>

In the GWT template, additional sentences of each kind are written in the next line and receive the prefix and.

Features and Scenarios must have a name, but their description are not required. Thus, you can write the above example like this (although we recommend against):

Feature: Sales Report

Scenario: Daily sales report

Feature files receive the extension .feature. Only one feature per file is allowed.

Variants

A Variant is a test-oriented declaration that describes a possible interaction between a user and the system in order to accomplish a Scenario. You can write more than one Variantfor a same Scenario .

A Variant must use the GWT template and first person singular ("I") in its sentences. Concordia Compiler uses Natural Language Processing (NLP) techniques to understand Variant sentences, aiming to transform them into test cases. Example:

Feature: New User

Scenario: Successful registration

  Variant: Register with a new account
    Given that I am on "https://example.com/register"
    When I fill <#name> with "Bob"
      and I fill <#dateOfBirth> with "11/30/1990"
      and I enter with "bob@example.com" in <#email>
      and I inform "m123456" in <#pass>    
      and I type "m123456" in <#confirmation>
      and I click <#ok>
    Then I see "Welcome, Bob"

Concordia Compiler understands the sentences from line 7 to11 as being variations of the same action, fill.

Of course, the vocabulary is limited. Although, you can extend it whether you need (by adding new synonyms to a specific dictionary file). Actions and common variations are documented and suggestions are always welcomed.

In the sentences above, the values were denoted between quotation marks, e.g. "bob@example.com". Numeric values are also accepted and do not need quotation marks. Example: 97.

Widgets (of the user interface) are written between < and >. Example: <#email>. Concordia adopts the well-known CSS query selectors to locate widgets, regardless the user interface (its plugins convert them to the format adopted by the testing framework). You can also use XPath (although some plug-ins may not accept it). Examples:

# (hashtag) to find a widget by its id. Example: #email
. (dot) to find a widget by its class, if applicable. Example: .colorful
@ (at) to find a widget by its name. Example: @pass
~ (tilde) to find a widget by its mobile name, if applicable. Example: ~pass
no prefixed characters, to find a widget by its type. Example: input
// (double slash) to find a widget by its XPath. Example: //table/tbody/tr[1]

Whether you are testing a web application, you may find Katalon-Concordia useful. It's a browser extension - available for Google Chrome and Mozilla Firefox - that extends Katalon Recorder to save a recorded interaction as Variant sentences. Using it may reduce the time needed to locate the widgets of an existing application.

States

A State is denoted by a text between tilde (~), such as ~a state~, and must be written inside Variant sentences only. Example:

Then I have a ~user logged in~

A Variant can both produce or require states. A State is produced by a Then sentence, like in the prior example. A State is required by a Given or a When sentence. For instance:

Given that I have ~user logged in~

By declaring a required State, you're establishing a dependency among the current Variant and the ones that can produce it. Since that State is often produced by a Variant from another Feature, you have to import the corresponding Feature file. Concordia Compiler will only search for states from imported Feature files.

When transforming Variants into Test Cases, Concordia Compiler will:

Remove any (Then) sentences with a produced State; and
Replace any (Given or When) sentences with a required State by sentences from the Variant that can produce it.

Whether there are different Variants that can produce a (same) required State, Concordia Compiler will be able to combine each of them with the current Variant, aiming to produce Test Cases that explore different execution paths. For more information, please read the section State Based-Combination.

Example

Let's say that we have a simple Login feature like this:

Feature: Login

Scenario: Sucessful login

  Variant: Login with username and password
    Given that I am on the [Login Screen]
    When I enter with "bob" in {Username}
      and I enter with "123456" in {Password}
      and I click on {OK}
    Then I see "Welcome"
      and I have ~user logged in~


Constants:
  - "Login Screen" is "/page"
  
UI Element: Username

UI Element: Password

UI Element: OK
  - type is button

Now let's suppose that a user has to be logged in to access its account details:

import "login.feature"

Feature: My Account

  Scenario: See my account data
  
    Variant: Show basic data by default
      Given that I have ~user logged in~
        and I am on the [Account Screen]
      Then I see {User} with "bob"
        and I see {Name} with "Robert Downey Jr"


Constants:
  - "Account Screen" is "/account"
        
UI Element: User

Concordia Compiler will produce my-account.testcase with a Test Case like the following, that includes steps from both the features:

import "my-account.feature"

@scenario(1)
@variant(1)
Test Case: See my account data - 1
  Given that I am on the "/login"
  When I enter with "bob" in <username>
    and I enter with "123456" in <password>
    and I click on <ok>
  Then I see "Welcome"
  Given that I am on the "/account
  Then I see <user> with "bob"
    and I see <name> with "Robert Downey Jr"

Concordia Compiler can optimize the combination among Features in order to reduce the amount of produced test cases and, therefore, the total execution time. You can also opt to get full coverage - which is ideal before releasing a feature.

Test Cases

A Test Case represents a test case produced for a certain Variant . Concordia Compiler can generate many Test Case declarations from a same Variant, each one covering different states, input combinations and oracles. The generation considers the following declarations:

Variant
states
UI Element
Constants
Table
Database

The simplest Test Case is one whose sentences are exactly equal to the Variant's. That's the case when the Variant does not use other declarations and it explicits all the input test data to use.

For comparison, whether we simply remove with "Bob"from the sentenceWhen I fill <#name> with "Bob" , Concordia Compiler will generate a pseudo-random value to complete the sentence, and it will be like this: When I fill <#name> with "A~!39esljdns so2u1 *ns%".

Concordia Compiler uses a single seed to generate pseudo-random numbers in its algorithms and values. Everytime it runs, a new seed is produced. You can set the seed using the CLI parameter --seed. By setting the seed, the compiler will pick the same paths and produce the same values - which is desirable for reproducing the same behavior in the application under test, but not for discovering new defects.

Don't worry about writing Test Cases. Write Variants and let Concordia generates the Test Cases for you.

UI Elements

A UI Element represents a user interface element and can denote its expected behavior through properties.

Although a Widget is easy to declare, it can have some shortcomings:

1) It can make a sentence hard to read, specially when it uses XPath or CSS locators.

For example, the sentence

When I click on <//*[@id="section1"]/div[1]/div[2]>

does inform what the action really means. The same sentence with a UI Element reference becomes

When I click on {Continue}

Then the UI Element can be declared as:

UI Element: Continue
  - locator: //*[@id="section1"]/div[1]/div[2]

Here locator is a property that denotes how to find it in the UI.

2. It can be hard to maintain. When a Widget appears in more than one Scenario and its locator needs to change (e.g. you have changed it in the UI), you need to search and replace all its occurrences in the document. With a UI Element, you just need to change the property locator.

3. It does not offer resources for test generation other than its locator.

A UI Element offers properties that can describe the behavior related to it. These properties are used to infer the test data and test oracles to generate.

Example:

UI Element: Net Price
  - minimum value is 10.00
    Otherwise I see "Net Price must be greater than U$ 10"

Concordia Compiler infers 6 possible test cases from the above declaration, applying different testing techniques.

These are the UI Element properties:

id or locator ⇨ how to locate the widget
type ⇨ widget type, like textbox, button, etc.
editable ⇨ whether the UI Element is editable (or not)
data type ⇨ data type of an editable UI Element
required⇨ whether the UI Element is required (or not)
format⇨ data format, denoted by a regular expression
value⇨ defines how the value is produced
minimum value ⇨ defines how the minimum value is produced
maximum value ⇨ defines how the maximum value is produced
minimum length ⇨ defines how the minimum length is produced
maximum length ⇨ defines how the maximum length is produced

By default, declaring a UI Element without properties is the same as declaring it with:

id or locator equal to the UI Element name in camelCase (e.g., "Net Price" becomes "netPrice"). The adopted case is configurable.
type equal to textbox
data type equal to string

Concordia Compiler is able to guess some properties from the others. For instance, editable is enabled when type is a widget capable of receiving input data; data type can be inferred from any attributed value; etc.

Another example:

UI Element: Profession
  - value is in [ "Programmer", "Tester", "Analyst" ]

Concordia Compiler infers 4 possible test cases from the above declaration, applying different testing techniques.

Please see the language reference for more details. Let's proceed with the other declarations for now.

Import

A Import declaration allows you to import declarations from another .featurefile. Example:

Import "login.feature"

Imported declarations are Feature, Constants, Table, Database, and UI Element.

Constants

A Constants block defines one or more constants. Every Constant holds an immutable value that is accessible through the declared name. Example:

Constants:
  - "Register Page" is "https://example.com/register"
  - "Coupon Number" is 12345

Constants are accessed by their name between [ and ]. Example:

    Given that I am on the [Register Page]
    When I fill {Coupon} with [Coupon Number]

Constants are global declarations and cannot have duplicate names. You can use them in UI Element properties and Variant sentences.

Table

A Table declares values in rows and columns to use in UI Element properties. Example:

Table: Customer
  | name           | age |
  | Ada Lovelace   | 36  |
  | Dennis Ritchie | 70  |
  | Donald Knuth   | 82  |
  | George Boole   | 49  |
  | Niklaus Wirth  | 83  |
  
  
UI Element: Name
  - value comes from "SELECT name FROM [Customer]"
    Otherwise I see "Customer not found."

Concordia Compiler infers 4 possible test cases from the above declaration, applying different testing techniques.

Note that SQL is used to select values from the declared table, which is denoted between [ and ] (like Constants). Table names are global and share the same namespace as Constant names.

Database

A Database provides a way to indicate a connection to an existing database or file. Example:

Database: AcmeDB
  - type is "mysql"
  - name is "acme"
  - host is "http://127.0.0.1"
  - username is "tester"
  - password is "testing123"
  
  UI Element: Name
  - value comes from "SELECT name FROM [AcmeDB.customer]"
    Otherwise I see "Customer not found."

Concordia Compiler infers 4 possible test cases from the above declaration, applying different testing techniques.

SQL is used to select values from a table or view from the declared database, which is denoted between [ and ] . Its table or view names are separated by a dot (.). Database names are global and share the same namespace as Tables and Constants.

Test Events

In order to set the execution environment up, testers can define events that can run database commands or command-line scripts. These events can occur before or after Scenarios or Features:

Before Each Scenario: executes before every test of each Scenario of the current Feature.
After Each Scenario: executes after every test of each Scenario of the current Feature.
Before Feature: executes once, before all the tests of the current Feature.
After Feature: executes once, after all the tests of the current Feature.
Before All: executes once, before all the tests of the application under test.
After All: executes once, after all the tests of the application under test.

Test Events are written like Variants, using Given-When-Then. Since they usually perform actions, it is very likely to useWhen in most sentences. Example:

Before Feature:
   Given that I connect to [AcmeDB]
   When I run the script 'DELETE FROM customer'
     and I run the script 'INSERT INTO customer ( name, age ) VALUES ( "Alice", 20 ), ( "Bob", 30 )'     
     and I run the command 'rmdir log'

Commands and database scripts are denoted between apostrophe (').

Concordia

Quick Start

System Requirements

Step 0: Initialize

Step 1: Install

Step 2: Configure

Step 3: Create a feature

Step 4: Execute

Notes on installation options

Local installation (recommended)

Introduction

Understanding Concordia

Concordia metalanguage

Concordia compiler

Concordia plug-ins

A Small Example

Execution without generation

Output

3. Analyze the results

See also

Using a Plug-in

Plug-ins

Installation

Upgrade

How to use a plug-in

Commonly used commands

Starting a testing server

Generating and executing test scripts

Generating test scripts without executing them

Executing existing test scripts (without generating them)

See also

Using Databases

Installation

Installing

Uninstalling

Upgrading

Usage

See also

Related things

Tools

Publications

Language

Concordia Language

Line comments

Common line comment

Special line comment

Tag

Special tag

Common tag

Feature

State

Scenario

Variant

Best practices

Widgets

Constants

Table

Database

UI Element

Import

Test Case

Test Events

Actions

accept

amOn

append

attachFile

cancel

check

clear

click

close

connect

disconnect

doubleClick

drag

fill

fill + target

hide

install

`accept`

`amOn`

`append`

`attachFile`

`cancel`

`check`

`clear`

`click`

`close`

`connect`

`disconnect`

`doubleClick`

`drag`

`fill`

`hide`

`install`

`maximize`

`move`

`open`

`press`

`pull`

`refresh`

`remove`

`resize`

`rightClick`

`run`

`saveScreenshot`

`scrollTo`

`see`

`select`

`shake`

`swipe`

`switch`

`tap`

`uncheck`

`wait`

`accept`

`amOn`

`append`

`attachFile`

`cancel`

`check`

`clear`

`click`

`close`

`connect`

`disconnect`

`doubleClick`

`drag`

`fill`