Overview of the Concordia declarations:
A Featuredefines a desired behavior for the software, a piece of functionality that gives some business value. A Scenario is a high-level usage scenario of a Feature. A Feature can have many Scenarios.
Since Feature and Scenario are business-oriented declarations, they are not used by Concordia Compiler to generate test cases. You can write them freely. Although, we recommend you to use the User Story template for a Feature and the Given-When-Then template for a Scenario. Example:
This is the User Story template:
As a <role>
I would like to <goal to perform>
In order to <received benefit> (or So that <received benefit>)
And this is the Given-When-Then (GWT) template:
Given <some context or precondition>
When <some action is carried out>
Then <a result or postcondition is observed>
In the GWT template, additional sentences of each kind are written in the next line and receive the prefix and.
Features and Scenarios must have a name, but their description are not required. Thus, you can write the above example like this (although we recommend against):
Feature files receive the extension .feature. Only one feature per file is allowed.
A Variant is a test-oriented declaration that describes a possible interaction between a user and the system in order to accomplish a Scenario. You can write more than one Variantfor a same Scenario .
A Variant must use the GWT template and first person singular ("I") in its sentences. Concordia Compiler uses Natural Language Processing (NLP) techniques to understand Variant sentences, aiming to transform them into test cases. Example:
Concordia Compiler understands the sentences from line 7 to11 as being variations of the same action, fill.
Of course, the vocabulary is limited. Although, you can extend it whether you need (by adding new synonyms to a specific dictionary file). Actions and common variations are documented and suggestions are always welcomed.
In the sentences above, the values were denoted between quotation marks, e.g. "bob@example.com". Numeric values are also accepted and do not need quotation marks. Example: 97.
Widgets (of the user interface) are written between < and >. Example: <#email>. Concordia adopts the well-known CSS query selectors to locate widgets, regardless the user interface (its plugins convert them to the format adopted by the testing framework). You can also use XPath (although some plug-ins may not accept it). Examples:
# (hashtag) to find a widget by its id. Example: #email
. (dot) to find a widget by its class, if applicable. Example: .colorful
@ (at) to find a widget by its name. Example: @pass
~ (tilde) to find a widget by its mobile name, if applicable. Example: ~pass
no prefixed characters, to find a widget by its type. Example: input
// (double slash) to find a widget by its XPath. Example: //table/tbody/tr[1]
Whether you are testing a web application, you may find Katalon-Concordia useful. It's a browser extension - available for Google Chrome and Mozilla Firefox - that extends Katalon Recorder to save a recorded interaction as Variant sentences. Using it may reduce the time needed to locate the widgets of an existing application.
A State is denoted by a text between tilde (~), such as ~a state~, and must be written inside Variant sentences only. Example:
A Variant can both produce or require states. A State is produced by a Then sentence, like in the prior example. A State is required by a Given or a When sentence. For instance:
By declaring a required State, you're establishing a dependency among the current Variant and the ones that can produce it. Since that State is often produced by a Variant from another Feature, you have to import the corresponding Feature file. Concordia Compiler will only search for states from imported Feature files.
When transforming Variants into Test Cases, Concordia Compiler will:
Remove any (Then) sentences with a produced State; and
Replace any (Given or When) sentences with a required State by sentences from the Variant that can produce it.
Whether there are different Variants that can produce a (same) required State, Concordia Compiler will be able to combine each of them with the current Variant, aiming to produce Test Cases that explore different execution paths. For more information, please read the section State Based-Combination.
Let's say that we have a simple Login feature like this:
Now let's suppose that a user has to be logged in to access its account details:
Concordia Compiler will produce my-account.testcase with a Test Case like the following, that includes steps from both the features:
Concordia Compiler can optimize the combination among Features in order to reduce the amount of produced test cases and, therefore, the total execution time. You can also opt to get full coverage - which is ideal before releasing a feature.
A Test Case represents a test case produced for a certain Variant . Concordia Compiler can generate many Test Case declarations from a same Variant, each one covering different states, input combinations and oracles. The generation considers the following declarations:
Variant
states
UI Element
Constants
Table
Database
The simplest Test Case is one whose sentences are exactly equal to the Variant's. That's the case when the Variant does not use other declarations and it explicits all the input test data to use.
For comparison, whether we simply remove with "Bob"from the sentenceWhen I fill <#name> with "Bob" , Concordia Compiler will generate a pseudo-random value to complete the sentence, and it will be like this: When I fill <#name> with "A~!39esljdns so2u1 *ns%".
Concordia Compiler uses a single seed to generate pseudo-random numbers in its algorithms and values. Everytime it runs, a new seed is produced. You can set the seed using the CLI parameter --seed. By setting the seed, the compiler will pick the same paths and produce the same values - which is desirable for reproducing the same behavior in the application under test, but not for discovering new defects.
Don't worry about writing Test Cases. Write Variants and let Concordia generates the Test Cases for you.
A UI Element represents a user interface element and can denote its expected behavior through properties.
Although a Widget is easy to declare, it can have some shortcomings:
1) It can make a sentence hard to read, specially when it uses XPath or CSS locators.
For example, the sentence
does inform what the action really means. The same sentence with a UI Element reference becomes
Then the UI Element can be declared as:
Here locator is a property that denotes how to find it in the UI.
2. It can be hard to maintain. When a Widget appears in more than one Scenario and its locator needs to change (e.g. you have changed it in the UI), you need to search and replace all its occurrences in the document. With a UI Element, you just need to change the property locator.
3. It does not offer resources for test generation other than its locator.
A UI Element offers properties that can describe the behavior related to it. These properties are used to infer the test data and test oracles to generate.
Example:
Concordia Compiler infers 6 possible test cases from the above declaration, applying different testing techniques.
These are the UI Element properties:
id or locator ⇨ how to locate the widget
type ⇨ widget type, like textbox, button, etc.
editable ⇨ whether the UI Element is editable (or not)
data type ⇨ data type of an editable UI Element
required⇨ whether the UI Element is required (or not)
format⇨ data format, denoted by a regular expression
value⇨ defines how the value is produced
minimum value ⇨ defines how the minimum value is produced
maximum value ⇨ defines how the maximum value is produced
minimum length ⇨ defines how the minimum length is produced
maximum length ⇨ defines how the maximum length is produced
By default, declaring a UI Element without properties is the same as declaring it with:
id or locator equal to the UI Element name in camelCase (e.g., "Net Price" becomes "netPrice"). The adopted case is configurable.
type equal to textbox
data type equal to string
Concordia Compiler is able to guess some properties from the others. For instance, editable is enabled when type is a widget capable of receiving input data; data type can be inferred from any attributed value; etc.
Another example:
Concordia Compiler infers 4 possible test cases from the above declaration, applying different testing techniques.
Please see the language reference for more details. Let's proceed with the other declarations for now.
A Import declaration allows you to import declarations from another .featurefile. Example:
Imported declarations are Feature, Constants, Table, Database, and UI Element.
A Constants block defines one or more constants. Every Constant holds an immutable value that is accessible through the declared name. Example:
Constants are accessed by their name between [ and ]. Example:
Constants are global declarations and cannot have duplicate names. You can use them in UI Element properties and Variant sentences.
A Table declares values in rows and columns to use in UI Element properties. Example:
Concordia Compiler infers 4 possible test cases from the above declaration, applying different testing techniques.
Note that SQL is used to select values from the declared table, which is denoted between [ and ] (like Constants). Table names are global and share the same namespace as Constant names.
A Database provides a way to indicate a connection to an existing database or file. Example:
Concordia Compiler infers 4 possible test cases from the above declaration, applying different testing techniques.
SQL is used to select values from a table or view from the declared database, which is denoted between [ and ] . Its table or view names are separated by a dot (.). Database names are global and share the same namespace as Tables and Constants.
In order to set the execution environment up, testers can define events that can run database commands or command-line scripts. These events can occur before or after Scenarios or Features:
Before Each Scenario: executes before every test of each Scenario of the current Feature.
After Each Scenario: executes after every test of each Scenario of the current Feature.
Before Feature: executes once, before all the tests of the current Feature.
After Feature: executes once, after all the tests of the current Feature.
Before All: executes once, before all the tests of the application under test.
After All: executes once, after all the tests of the application under test.
Test Events are written like Variants, using Given-When-Then. Since they usually perform actions, it is very likely to useWhen in most sentences. Example:
Commands and database scripts are denoted between apostrophe (').