Versioning

Concordia Compiler's versions are based on Semantic Versioning.

Although Semantic Versioning is conceived for APIs instead of for applications, we adopt a very similar convention. Thus, changes become predictable and you can know, from the version numbers, when a version is no more compatible with a previous version.

Given a version MAJOR.MINOR.UPDATE:

• MAIOR is increased when the Compiler or the Language is no more compatible with the previous version.

• MINOR is increased when adding functionality in a backwards-compatible manner.

• UPDATE is increased when there are fixes, little changes or little novelties - all of them backwards-compatible.

Examples:

• 0.2.0 is compatible with 0.1.0

• 0.1.1 is compatible with 0.1.0

• 1.0.0 is

Upgrade

To upgrade a local installation:

To upgrade a global installation:

Overview of the Concordia declarations:

Features and Scenarios

A Featuredefines a desired behavior for the software, a piece of functionality that gives some business value. A Scenario is a high-level usage scenario of a Feature. A Feature can have many Scenarios.

Since Feature and Scenario are business-oriented declarations, they are not used by Concordia Compiler to generate test cases. You can write them freely. Although, we recommend you to use the template for a Feature and the template for a Scenario. Example:

This is the template:

• As a <role>

• I would like to <goal to perform>

• In order to <received benefit> (or So that <received benefit>)

And this is the (GWT) template:

• Given <some context or precondition>

• When <some action is carried out>

• Then <a result or postcondition is observed>

In the GWT template, additional sentences of each kind are written in the next line and receive the prefix and.

Features and Scenarios must have a name, but their description are not required. Thus, you can write the above example like this (although we recommend against):

Feature files receive the extension .feature. Only one feature per file is allowed.

Variants

A Variant is a test-oriented declaration that describes a possible interaction between a user and the system in order to accomplish a Scenario. You can write more than one Variantfor a same Scenario .

A Variant must use the GWT template and first person singular ("I") in its sentences. Concordia Compiler uses techniques to understand Variant sentences, aiming to transform them into test cases. Example:

Concordia Compiler understands the sentences from line 7 to11 as being variations of the same , fill.

Of course, the vocabulary is limited. Although, you can extend it whether you need (by adding new synonyms to a specific dictionary file). Actions and common variations and suggestions are always .

In the sentences above, the values were denoted between quotation marks, e.g. "bob@example.com". Numeric values are also accepted and do not need quotation marks. Example: 97.

Widgets (of the user interface) are written between < and >. Example: <#email>. Concordia adopts the well-known to locate widgets, regardless the user interface (its plugins convert them to the format adopted by the testing framework). You can also use (although some plug-ins may not accept it). Examples:

• # (hashtag) to find a widget by its id. Example: #email

• . (dot) to find a widget by its class, if applicable. Example: .colorful

• @

States

A State is denoted by a text between tilde (~), such as ~a state~, and must be written inside sentences only. Example:

A can both produce or require states. A State is produced by a Then sentence, like in the prior example. A State is required by a Given or a When sentence. For instance:

By declaring a required State, you're establishing a dependency among the current and the ones that can produce it. Since that State is often produced by a Variant from another Feature, you have to the corresponding Feature file. Concordia Compiler will only search for states from imported Feature files.

When transforming Variants into , Concordia Compiler will:

• Remove any (Then) sentences with a produced State; and

• Replace any (Given or When) sentences with a required State by sentences from the Variant that can produce it.

Whether there are different s that can produce a (same) required State, Concordia Compiler will be able to combine each of them with the current Variant, aiming to produce that explore different execution paths. For more information, please read the section .

Example

Let's say that we have a simple Login feature like this:

Now let's suppose that a user has to be logged in to access its account details:

Concordia Compiler will produce my-account.testcase with a like the following, that includes steps from both the features:

Test Cases

A Test Case represents a test case produced for a certain Variant . Concordia Compiler can generate many Test Case declarations from a same Variant, each one covering different states, input combinations and oracles. The generation considers the following declarations:

• Variant

• states

• UI Element

• Constants

The simplest Test Case is one whose sentences are exactly equal to the Variant's. That's the case when the Variant does not use other declarations and it explicits all the input test data to use.

For comparison, whether we simply remove with "Bob"from the sentenceWhen I fill <#name> with "Bob" , Concordia Compiler will generate a pseudo-random value to complete the sentence, and it will be like this: When I fill <#name> with "A~!39esljdns so2u1 *ns%".

Concordia Compiler uses a single seed to generate pseudo-random numbers in its algorithms and values. Everytime it runs, a new seed is produced. You can set the seed using the CLI parameter --seed. By setting the seed, the compiler will pick the same paths and produce the same values - which is desirable for reproducing the same behavior in the application under test, but not for discovering new defects.

UI Elements

A UI Element represents a user interface element and can denote its expected behavior through properties.

Although a Widget is easy to declare, it can have some shortcomings:

1) It can make a sentence hard to read, specially when it uses XPath or CSS locators.

2. It can be hard to maintain. When a Widget appears in more than one Scenario and its locator needs to change (e.g. you have changed it in the UI), you need to search and replace all its occurrences in the document. With a UI Element, you just need to change the property locator.

3. It does not offer resources for test generation other than its locator.

A UI Element offers properties that can describe the behavior related to it. These properties are used to infer the test data and test oracles to generate.

Example:

These are the UI Element properties:

• id or locator ⇨ how to locate the widget

• type ⇨ widget type, like textbox, button, etc.

By default, declaring a UI Element without properties is the same as declaring it with:

• id or locator equal to the UI Element name in camelCase (e.g., "Net Price" becomes "netPrice"). The adopted case is configurable.

• type equal to textbox

• data type

Another example:

Please see the language reference for more details. Let's proceed with the other declarations for now.

Import

A Import declaration allows you to import declarations from another .featurefile. Example:

Imported declarations are Feature, Constants, Table, Database, and UI Element.

Constants

A Constants block defines one or more constants. Every Constant holds an immutable value that is accessible through the declared name. Example:

Constants are accessed by their name between [ and ]. Example:

Constants are global declarations and cannot have duplicate names. You can use them in UI Element properties and Variant sentences.

Table

A Table declares values in rows and columns to use in UI Element properties. Example:

Note that is used to select values from the declared table, which is denoted between [ and ] (like Constants). Table names are global and share the same namespace as Constant names.

Database

A Database provides a way to indicate a connection to an existing . Example:

is used to select values from a table or view from the declared database, which is denoted between [ and ] . Its table or view names are separated by a dot (.). Database names are global and share the same namespace as Tables and Constants.

Test Events

In order to set the execution environment up, testers can define events that can run database commands or command-line scripts. These events can occur before or after Scenarios or Features:

• Before Each Scenario: executes before every test of each Scenario of the current Feature.

• After Each Scenario: executes after every test of each Scenario of the current Feature.

• Before Feature: executes once, before all the tests of the current Feature.

Test Events are written like Variants, using Given-When-Then. Since they usually perform actions, it is very likely to useWhen in most sentences. Example:

Commands and database scripts are denoted between apostrophe (').

Imagine a solution in which you can write software requirements using an Agile-style, restricted natural language and it automatically generates for you using effective testing techniques such as , , and and it can also use test data from . Well, that's Concordia. Its solution is composed by a metalanguage and a compiler.

Concordia Compiler uses plug-ins for:

1. setting the testing environment up;

2. generating and executing test scripts;

3. converting and reporting test results.

Every plug-in can generate test scripts for a different testing framework or programming language.

Plug-ins

Notes:

• (*) RTS means "Requires a ", that is, if it requires a testing server to run the test scripts (e.g., Selenium Standalone).

• (**) Appium requires Java for testing web-based mobile applications or using the Android SDK.

• codeceptjs-testcafe uses the frameworks and , and works with probably any browser.

Installation

Use --plugin-install plus the plug-in name. Example:

Upgrade

Just uninstall the plug-in and then install it again. Example:

How to use a plug-in

Concordia has the following plug-in commands:

• plugin-install to install a plug-in

• plugin-uninstallto uninstall a plug-in

• plugin-serve to start a testing server using the plug-in

All but the latter command (plugin-list) require a plug-in name.

👉 Whether you have a with the property plugin defined, you can omit the plugin name from a command. Example:

Commonly used commands

Starting a testing server

Some plug-ins (WebDriverIO, Appium) require a testing server to execute test scripts. A testing server controls a browser or an emulator during tests.

We recommend to open a new terminal/console and then start the testing server:

The testing server will remain open. To stop it later, type Ctrl + C.

Generating and executing test scripts

Generating test scripts without executing them

You can use --no-run to avoid running test scripts and use --no-result to avoid reading the last report with test results. Test scripts will be generated but not executed.

Executing existing test scripts (without generating them)

You can use --no-script to avoid generating test scripts. Only existing test scripts will be executed.

See also

Welcome to the Concordia 2 documentation! This page will help you to get started fast. If you run into problems, you can find help on our .

System Requirements

⚠ UNDER CONSTRUCTION ⚠

In software testing, test coverage is some kind of metric that helps you to understand what parts of your application (or code) are exercised by your tests. The number of feasible paths through them grows exponentially with an increase in application size and can even be infinite in the case of applications with unbounded loop iterations. That is a problem called path explosion problem. Concordia Compiler deals with it by providing sets of combination and selection strategies, and trying to achieve full path coverage over time.

Test Coverage

• All the Features, Scenarios, and Variants are covered by default.

• The CLI parameter --files can filter the .feature files to be considered.

• The CLI parameter --ignore can indicate the .feature files to be ignored, when a directory is given.

State-based Strategies

In Concordia, you can declare a State in a Variant sentence using a text between tile (~), like this:

There are three types of State:

1. Precondition: when declared in a Given sentence;

2. State Call: when declared in a When sentence;

3. Postcondition: when declared in a Then sentence.

Both Preconditions and State Calls are considered required States. That is, they denote a dependency of a certain State of the system that needs to be executed. A Precondition needs to be executed before the Variant starts, and a State Call needs to be executed during the Variant's execution.

A Postcondition is a produced State, that is, a state of the system produced by a successful execution of a Variant. Therefore, whether something goes wrong during a Variant's execution, it will not produce the declared State.

When the current Variant requires a State, Concordia Compiler will look for imported Features' Variants able to produce it. To generate complete Test Cases for the current Variant, it will:

1. Select the Variants to combine;

2. Generate successful test scenarios for the selected Variants;

3. Select the successful test scenarios to combine;

4. Generate (successful and unsuccessful) test scenarios for the current Variant;

Steps 1 and 3 can adopt different strategies. Concordia Compiler let's you:

• Parameterize how the Variants will be selected, using --comb-variant; and

• Parameterize how the successful test scenarios will be combined, using --comb-state.

Variant selection

Available strategies for --comb-variant:

• random: Selects a random Variant that produces the required State. That's the default behavior;

• first: Selects the first Variant that produces the required State;

• fmi: Selects the first most important Variant (since two Variants can have the same importance value) that produces the required State;

Example:

State combination

Available strategies for --comb-state:

• sre: Single random of each - that is, randomly selects a single, successful test scenario of each selected Variant. That's the default behavior;

• sow : Shuffled one-wise - that is, shuffles the successful test scenarios than uses combination.

• ow

Example:

Full vs random selection

Strategies that use random selection can take different paths every time they are used. Furthermore, they reduce considerably the amount of generated paths - i.e., it avoids "path explosion" - and thus the amount of produced test cases.

Full-selection strategies can be used for increase path coverage. Although, it also increases the needed time to check all the paths, which may be undesirable for frequent tests.

By default, Concordia Compiler uses random selection strategies.

Testing Techniques

Concordia Compiler can infer input test data from Variants, UI Elements, Constants, Tables, and Databases. The more constraints you declare, the more test cases it generates.

Adopted techniques to generate input test data include:

These are well-known, effective black-box testing techniques for discovering relevant defects on applications.

Data Test Cases

We call Data Test Cases those test cases used to generate input test data. They are classified into the following groups: RANGE, LENGTH, FORMAT, SET, REQUIRED, and COMPUTED. The group COMPUTEDis not available on purpose, since a user-defined algorithm to produce test data can have bugs on itself. Thus, one should provide expected input and output values in order to check whether the application is able to correctly compute the output value based on the received input value.

Every group has a set of related data test cases, applied according to the declared constraints and selected algorithms:

Maximum length for random string values

By default, the maximum length for randomly-generated string values is 500. This value is used for reducing the time to run test scripts, since long strings take time to be entered.

You can set maximum length using the CLI parameter --random-max-string-size. Example:

You can also set it in the configuration file (.concordiarc) by adding the property randomMaxStringSize. Example:

Properties vs Data Test Cases

Data Test Cases (DTC) are selected for every declared and its properties. The more properties you declare for a UI Element, the more data you provide for Concordia Compiler to generate DTC.

Example of some evalutations:

• When no properties are declared, FILLED and NOT_FILLED are both applied and considered as valid values;

• When the property required is declared, NOT_FILLED (empty) is considered as an invalid value;

• When the property value

Example 1

Let's describe a named Salary :

When no property is defined or only the property data typeis defined,

We defined the property data type as double, since the default data type is string.

Since few restrictions were made, Salary will be tested with the test cases of the group REQUIRED:

1. FILLED: a pseudo-random double value is generated;

2. NOT_FILLED: an empty value will be used.

Now let's add a minimum value restriction.

Some tests of the group RANGE are now applicable:

1. LOWEST_VALUE: the lowest possible double is used

2. RANDOM_BELOW_MIN_VALUE: a random double before the minimum value is generated

3. JUST_BELOW_MIN_VALUE: a double just below the minimum value is used (e.g., 999.99)

Since 1000.00 is the minimum value, the data produced by the tests 1, 2, 3, and 6 of the group VALUE are considered invalid, while 4 and 5 are not. For these tests considered invalid, the behavior defined in Otherwise, that is

is expected to happen. In other words, this behavior serves as and must occur only when the produced value is invalid.

Unlike this example, when the expected system behavior for invalid values is not specified and a test data is considered invalid, Concordia expects that test should fail. In this case, it generates the Test Case with the tag @fail.

Now let's add maximum value restriction:

All the tests of the group RANGE are now applicable. That is, the following tests will be included:

1. MEDIAN_VALUE: the median between the minimum and the maximum values

2. RANDOM_BETWEEN_MIN_MAX_VALUES: a pseudo-random double value between the minimum and the maximum values

3. JUST_BELOW_MAX_VALUE: the value just below the maximum value

The tests from 5 to 7 will produce values considered invalid.

Example 2

Let's define a user interface element named Profession and a table named Professions from which the values come from:

Applicable test are:

• FILLED

• NOT_FILLED

• FIRST_ELEMENT

The first two tests are in the group REQUIRED. Since we declared Profession as having a required value, the test FILLED is considered valid but NOT_FILLED is not. Therefore, it is important to remember declaring required inputs accordingly.

The last four tests are in the group SET. Only the last one, NOT_IN_SET, will produce a value considered invalid.

Example 3

In this example, let's adjust the past two examples to make Salary rules dynamic and change according to the Profession.

Firstly, we add two columns the the Professions table:

Then, we change the rules to retrieve the values from the table:

The reference to the UI Element {Profession} inside the query, makes the rules of Salary depend on Profession. Every time a Profession is selected, the minimum value and the maximum value of Salary changes according to the columns min_salary and max_salary of the table Professions.

Concordia Compiler can use a configuration file in named .concordiarc. You can generate it by running the following command:

You'll be asked about the desired configuration and then the file will be saved.

Example:

Configuration file's properties are similar to CLI parameters. The example above defines: "pt" (meaning "Portuguese") as the default language; the directory "docs/features" for features and test cases; the plug-in "codeceptjs-testcafe" for generating test scripts; the directory "test/e2e" for test scripts; and the directory

Concordia Language contains a set of meta-actions that can be used by different testing frameworks for generating test scripts. Concordia Compiler uses Intent Recognition (yes, like chatbots do) to understand sentences and extract the desired action to perform. Later these actions are sent to testing frameworks for conversion into commands.

The following list presents the available actions and gives some examples on how to use them in sentences. We expect that you do not have to memorize them but, instead, see the things you can do. Try to write the sentences the way you speak, using first person singular (I

Migrations are only needed when upgrading to a major version. See Versions and Upgrade for more information.

To upgrade to a major version:

1. Uninstall the current version, e.g.: npm uninstall -D concordialang

2. Install the new version, e.g.: npm install -D concordialang

3. Now proceed as described below (depending on your current version).

Note: For a global installation, replace -D with -g.

From 1.x to 2.x

1. Update your plug-in

Concordia 2 has a new plugin API, not compatible with plug-ins for Concordia 1. So uninstall your current plug-in and install a new one. Example:

Alternatively, you can install one of the then change your configuration file to use it. Example:

2. Install database drivers

Whether your Concordia specification access a database, you have to install the corresponding database driver. Concordia 2 does not install them by default anymore, as Concordia 1 did, aiming to reduce the amount of downloaded dependencies.

Concordia 2 has a new CLI option --db-install to help with that. Example:

See to install the proper driver(s) for your application.

From 0.x to 1.x

1. Update your configuration file, if needed

Whether you project has a configuration file .concordiarc, open it with a text editor. If the file has a property "plugin" with the value "codeceptjs", you must change it to "codeceptjs-webdriverio".

2. Install the new plug-in

You can install any of the , currently codeceptjs-webdriverio or codeceptjs-appium. Example:

See also

Work in progress and issues:

• Kanban Board (GitHub)

• Issues (GitHub)

Logo

Yes, this project does not have a logo yet 😱. Designers interested in helping us (for free 🙏), please contact us on our or .

The logo should reflect the idea behind the language:

The tool generates test cases and test scripts that verify if the application under test meets the specified functional requirements. Therefore, the idea of "verification" or "testing" could be in the logo.

Plug-ins

Interested? See .

👉 Our next goal is a plugin for .

Suggestions:

• JavaScript and friends: (web), (web, mobile, desktop)

• PHP: (web)

• Java: (web), (android), (desktop)

Language

Variant Background

A Variant Background defines steps that must be executed before all the Variants of a certain Scenario. Example:

MOTIVATION: It allows to keep initial, repeated steps in a common place.

Constant names without quotation marks

Like this:

MOTIVATION: Simpler syntax.

Visual comparison

Please see .

MOTIVATION: It allows to perform visual comparison and detect related bugs.

File content comparison

To use the characters """ (three quotation marks) to denote a text content to be compared. Example:

MOTIVATION: It facilitates comparisons with text files or multiple-line strings.

Table matching

To allow a given UI Element or UI Literal to match a certain Table.

Example 1:

Example 2:

In which Some Table is declared like this:

It should make target table's rows to match the declared ones.

MOTIVATION: It facilitates the verification of expected data in tables from a user interface.

NOTES: Probably it requires to convert the Concordia Action to many abstract commands that verify every table row independently. Usually testing frameworks do not provide such kind of verification (for many rows at once).

Date and time expressions inside tables

Use Concordia 2 data and time expressions inside table rows. They must be written between ` or some other character. Example:

MOTIVATION: Concordia Language allows date and time expressions inside UI Properties. Using them inside tables can be useful.

Dynamic states

States that vary according to some generated value. Example:

According to the selected user, it will produce a different State. For example, when "bob" is selected, the produced state will be admin is logged in and when "joe" is selected, the produced state will be guest is logged in .

Thus, Features could depend on static or dynamic states.

MOTIVATION: Making states more flexible.

Alternative states

To provide a Given sentence that requires one - and only one - of the provided states. Example:

MOTIVATION: It allows to perform a XOR operation with the given states using natural language, aiming at choosing one and only one of the given states.

Annotations to parameterize combination strategies

To provide annotations parameterize a test combination strategy for a specific Feature.

For instance, to configure the combination of invalid values for a specific Feature:

The annotations should correspond to the CLI parameters.

MOTIVATION: Increase the flexibility of the test generation procedure.

Multi-line queries

Currently:

Proposal (to accept as valid):

Alternative proposal:

MOTIVATION: Make it easier to write/read SQL statements.

Compiler

Selection Options

• Select a Feature for Test Case generation, without having to include its dependencies:

  • by importance value

• Select a Scenario for Test Case generation:

Watch Mode

• Provide the option --watch to generate .testcase files when their corresponding .feature file changes or an imported .feature file changes. A new seed must be produced, except when explicitly provided (using --seed or configuration file).

Language Support

Concordia has language constructions that the Compiler does not support yet. Examples:

• Variant Background: Implement Variant Background, which is part of the Concordia Language but was not implemented yet by the Concordia Compiler.

• Support the tag in Features and Scenarios. Currently it is supported inVariants and Test Cases. Whether added to a Feature or a Scenario, it would not generate Test Cases.

New Test Cases

• Test cases that explore SQL Injection: Using declared Databases and Queries to formulate Test Cases that generate strings with SQL Injection commands.

• Test cases that use naughty strings as test data: Using a as test data.

UI and Report

To provide the CLI parameter --report for indicating the corresponding plug-in. Example:

which should be equivalent to

Tool integration

Integration with text editors

• Create new projects for auto-completion plug-ins for text editors such as VS Code, Sublime Text, Atom, etc. Example for VS Code: .

• Be able to run test scripts using Concordia from the IDE.

Integration with reporting tools

• Create integration with . Other reporters (e.g., ) can be added further.

Performance improvements

Avoid generating test scripts when their Test Case files did not change

Keep some hash control or use Git information when available. Hashes can be stored in a .json or .yml file. Example:

Generate unrelated files in parallel

NodeJS has made considerable progress since version 10.5 adding support to . Stable support for Worker Threads was added in version 12 LTS. Although, it would increase minimum requirements to install Concordia Compiler - currently it requires NodeJS 8.0.

Tools

• Katalon-Concordia is a browser extension for Chrome and Firefox that converts recordings from Katalon Recorder (which is a capture-replay tool - that is, it records all the actions that you perform while interacting with the application under test in order the reproduce them later) to Variant sentences. That's very handful for discovering elements' identification in web applications (e.g., their id properties or their ) and obtaining Variant sentences with the corresponding interaction.

• is a command-line tool that generates User Interface Prototypes (UIP) from Concordia features. It can help discussing features with stakeholders and accelerating their construction. Currently there is a customizable plug-in for HTML-based interfaces.

• is a command-line tool that generates an HTML page with an interactive graph that shows features files' relationships.

Publications

1. PINTO, Thiago Delgado. Unifying Agile requirements specification quality control and implementation conformance assurance. 2018. Doctoral Thesis. Pontifical Catholic University of Rio de Janeiro, Informatics Department, 2018.

2. PINTO, Thiago Delgado; COSTA, Pablo Veiga; GONÇALVES, Willian Inácio. User Interface Prototype Generation from Agile Requirements Specifications written in Concordia. 2019. WebMedia'19: Proceedings on the 25th Brazilian Symposium on Multimedia and the Web, Rio de Janeiro, 2019. ACM. DOI:

Let's suppose that we have to create two simple features for a web application: "Login" and "Add Product". After discussing them with stakeholders and the team, we created the following Concordia specification (see the two tabs):

Feature: Login
  As a user
  I would like to authenticate myself
  In order to access the application

These features could correspond to the following sketches:

The specification helps stakeholders, analysis, testers and developers to create a shared understanding about the application and its business rules. It helps with the requirements validation, knowledge documentation, project design, implementation and testing.

Different from Use Cases and other requirements specification formats, with Concordia you probably don't need to define different scenarios for validation. Instead of having to manually define a set scenarios for validation, you just need to define how the UI elements should behavior - in their declaration - and the compiler will generate different scenarios for you in the form of Test Cases. Whether your application has a lot of such rules, your team will certainly benefit from it.

<TO-DO (UNDER CONSTRUCTION)>

that a Variant express a possible interaction between a user and the application in order to complete its (business-focused) Scenario. lt follows the syntax and always uses the word "I" to represent the user or user role denoted in the Feature's description.

Let's run Concordia Compiler with a to produce the same results over and over again: