# Test Coverage and Techniques **⚠ UNDER CONSTRUCTION ⚠** In software testing, **test coverage** is some kind of metric that helps you to understand what parts of your application (or code) are exercised by your tests. The number of feasible paths through them grows exponentially with an increase in application size and can even be infinite in the case of applications with unbounded loop iterations. That is a problem called **path explosion problem**. Concordia Compiler deals with it by providing sets of **combination and selection strategies**, and trying to achieve full path coverage **over time**. ## Test Coverage * All the Features, Scenarios, and Variants are covered by default. * The CLI parameter `--files` can filter the `.feature` files to be considered. * The CLI parameter `--ignore` can indicate the `.feature` files to be ignored, when a directory is given. * The [tag](https://concordialang.gitbook.io/concordialang/language/concordia#tag) `@ignore` can be used to mark a `Feature` or `Variant` to be ignored by the test generator. However, it can still be used by other Features or Variants. * ~~The~~ [~~tag~~](https://concordialang.gitbook.io/concordialang/language/concordia#tag) ~~`@importance` (*e.g.*, `@importance( 8 )`) can be used to denote the importance of a Feature. CLI parameters `--sel-min-feature` and `--sel-max-feature` can then be used to filter the features to be considered by the test generator. Example: `concordia --sel-min-feature 7` makes the compiler considers the features with importance value of `7` or above. By default, all the features receive an importance value of `5`.~~ * Variants are selected and combined using State-based Strategies (see below). * All the UI Elements constraints are covered by default, using a set of Testing Techniques (see below). ## State-based Strategies In Concordia, you can declare a State in a Variant sentence using a text between tile (`~`), like this: ``` Given that I have ~payment method selected~ ``` There are three types of State: 1. *Precondition*: when declared in a `Given` sentence; 2. *State Call*: when declared in a `When` sentence; 3. *Postcondition*: when declared in a `Then` sentence. Both *Preconditions* and State Calls are considered **required** States. That is, they denote a dependency of a certain State of the system that needs to be executed. A Precondition needs to be executed *before* the Variant starts, and a State Call needs to be executed *during* the Variant's execution. A Postcondition is a **produced** State, that is, a state of the system produced by a *successful* execution of a Variant. Therefore, whether something goes wrong during a Variant's execution, it will *not* produce the declared State. When the current Variant *requires* a State, Concordia Compiler will look for imported Features' Variants able to *produce* it. To generate complete Test Cases for the current Variant, it will: 1. Select the Variants to combine; 2. Generate successful test scenarios for the selected Variants; 3. Select the successful test scenarios to combine; 4. Generate (successful and unsuccessful) test scenarios for the *current* Variant; 5. Combine the selected successful test scenarios with the test scenarios of the current Variant; 6. Transform all the test scenarios into test cases (*i.e.*, valued test scenarios). Steps 1 and 3 can adopt different strategies. Concordia Compiler let's you: * Parameterize how the Variants will be selected, using `--comb-variant`; and * Parameterize how the successful test scenarios will be combined, using `--comb-state`. ### Variant selection Available strategies for `--comb-variant`: * `random`: Selects a random Variant that produces the required State. **That's the default behavior**; * `first`: Selects the first Variant that produces the required State; * `fmi`: Selects the first most important Variant (since two Variants can have the same importance value) that produces the required State; * `all`: Selects all the Variants that produce the required State. Example: ``` npx concordia --comb-variant=all ``` ### State combination Available strategies for `--comb-state`: * `sre`: *Single random of each* - that is, randomly selects a single, successful test scenario of each selected Variant. **That's the default behavior**; * `sow` : *Shuffled one-wise* - that is, shuffles the successful test scenarios than uses [one-wise](https://github.com/thiagodp/one-wise) combination. * `ow`: [One-wise](https://github.com/thiagodp/one-wise) selection; * `all`: Selects all the successful test scenarios to combine. Example: ``` npx concordia --comb-state=all ``` ### Full vs random selection Strategies that use *random selection* can take different paths every time they are used. Furthermore, they reduce considerably the amount of generated paths - *i.e.*, it avoids "path explosion" - and thus the amount of produced test cases. Full-selection strategies can be used for increase path coverage. Although, it also increases the needed time to check all the paths, which may be undesirable for frequent tests. By default, Concordia Compiler uses random selection strategies. ## Testing Techniques Concordia Compiler can infer input test data from Variants, UI Elements, Constants, Tables, and Databases. The more constraints you declare, the more test cases it generates. Adopted techniques to generate **input test data** include: * [Boundary-value analysis](https://en.wikipedia.org/wiki/Boundary-value_analysis) * [Equivalence partitioning](https://en.wikipedia.org/wiki/Equivalence_partitioning) * [Random testing](https://en.wikipedia.org/wiki/Random_testing) * [Special cases](https://en.wikipedia.org/wiki/Corner_case) These are well-known, effective black-box testing techniques for discovering relevant defects on applications. ### Data Test Cases We call *Data Test Cases* those test cases used to generate input test data. They are classified into the following **groups**: `RANGE`, `LENGTH`, `FORMAT`, `SET`, `REQUIRED`, and `COMPUTED`. The group `COMPUTED`is not available on purpose, since a user-defined algorithm to produce test data can have bugs on itself. Thus, one should provide expected **input** and **output** values in order to check whether the application is able to correctly compute the output value based on the received input value. Every group has a set of related data test cases, applied according to the declared constraints and selected algorithms: | Group | Data Test Case | Description | | ------------ | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------- | | RANGE | LOWEST\_VALUE | The lowest value for the data type, *e.g.*, lowest integer | | | RANDOM\_BELOW\_MIN\_VALUE | A random value below the minimum value | | | JUST\_BELOW\_MIN\_VALUE | The value just below the minimum value, considering the data type and decimal places if applicable | | | MIN\_VALUE | Exactly the minimum value | | | JUST\_ABOVE\_MIN\_VALUE | The value just above the minimum value, considering the data type and decimal places if applicable | | | ZERO\_VALUE | Zero (`0`) | | | MEDIAN\_VALUE | The median between the minimum value and the maximum value | | | RANDOM\_BETWEEN\_MIN\_MAX\_VALUES | A random value between the minimum value and the maximum value | | | JUST\_BELOW\_MAX\_VALUE | The value just below the maximum value, considering the data type and decimal places if applicable | | | MAX\_VALUE | Exactly the maximum value | | | JUST\_ABOVE\_MAX\_VALUE | The value just above the maximum value, considering the data type and decimal places if applicable | | | RANDOM\_ABOVE\_MAX\_VALUE | A random value above the maximum value | | | GREATEST\_VALUE | The greatest value for the data type, *e.g.*, greatest integer | | LENGTH | LOWEST\_LENGTH | An empty string | | | RANDOM\_BELOW\_MIN\_LENGTH | A string with random characters and random length, less than the minimum length | | | JUST\_BELOW\_MIN\_LENGTH | A string with random characters and length exactly below the minimum length | | | MIN\_LENGTH | A string with random characters and length exactly equal to the minimum length | | | JUST\_ABOVE\_MIN\_LENGTH | A string with random characters and length exactly above the minimum length | | | MEDIAN\_LENGTH | A string with random characters and length equal to the median between the minimum length and the maximum length | | | RANDOM\_BETWEEN\_MIN\_MAX\_LENGTHS | A string with random characters and random length, between the minimum length and the maximum length | | | JUST\_BELOW\_MAX\_LENGTH | A string with random characters and length exactly below the maximum length | | | MAX\_LENGTH | A string with random characters and length exactly equal to the maximum length | | | JUST\_ABOVE\_MAX\_LENGTH | A string with random characters and length exactly above the maximum length | | | RANDOM\_ABOVE\_MAX\_LENGTH | A string with random characters and random length, greater than the maximum length | | | GREATEST\_LENGTH | The greatest length supported for a string (*see Notes*) | | FORMAT | VALID\_FORMAT | A value that matches the defined regular expression | | | INVALID\_FORMAT | A value that does not match the defined regular expression | | SET | FIRST\_ELEMENT | The first element in the defined set or query result | | | RANDOM\_ELEMENT | A random element in the defined set or query result | | | LAST\_ELEMENT | The last element in the defined set or query result | | | NOT\_IN\_SET | A value that does not belong to the defined set or query result | | REQUIRED | FILLED | A random value | | | NOT\_FILLED | Empty value | | ~~COMPUTED~~ | ~~RIGHT\_COMPUTATION~~ | ~~A value generated by the defined algorithm~~ | | | ~~WRONG\_COMPUTATION~~ | ~~A value different from that generated by the defined algorithm~~ | ### ### **Maximum length for random string values** By default, the maximum length for randomly-generated string values is `500`. This value is used for reducing the time to run test scripts, since long strings take time to be entered. You can set maximum length using the CLI parameter `--random-max-string-size`. Example: ``` npx concordia --random-max-string-size=300 ``` You can also set it in the configuration file (`.concordiarc`) by adding the property `randomMaxStringSize`. Example: ```javascript "randomMaxStringSize": 300 ``` ### Properties vs Data Test Cases Data Test Cases (DTC) are selected for every declared [UI Element](https://concordialang.gitbook.io/concordialang/language/concordia#ui-element) and its properties. The more properties you declare for a UI Element, the more data you provide for Concordia Compiler to generate DTC. Example of *some* evalutations: * When **no properties** are declared, `FILLED` and `NOT_FILLED` are both applied and considered as valid values; * When the property **required** is declared, `NOT_FILLED` (empty) is considered as an invalid value; * When the property **value** is declared: * if it comes from a *set* of values (inclusing a query result), all the DTC of the group `SET` are applied; * otherwise, ... * When the property **minimum value** is declared: {% hint style="info" %} There is more logic involved for generating these values. ... {% endhint %} ### **Example 1** Let's describe a [user interface element](https://concordialang.gitbook.io/concordialang/how-it-works/broken-reference) named `Salary` : ``` UI Element: Salary - data type is double ``` When no property is defined or only the property `data type`is defined, We defined the property `data type` as `double`, since the default data type is `string`. Since few restrictions were made, `Salary` will be tested with the test cases of the group `REQUIRED`: 1. `FILLED`: a pseudo-random double value is generated; 2. `NOT_FILLED`: an empty value will be used. Now let's add a **minimum value** restriction. ``` UI Element: Salary - data type is double - minimum value is 1000.00 Otherwise I must see "Salary must be greater than or equal to 1000" ``` Some tests of the group `RANGE` are now applicable: 1. `LOWEST_VALUE`: the lowest possible double is used 2. `RANDOM_BELOW_MIN_VALUE`: a random double before the minimum value is generated 3. `JUST_BELOW_MIN_VALUE`: a double just below the minimum value is used (*e.g.*, `999.99`) 4. `MIN_VALUE`: the minimum value is used 5. `JUST_ABOVE_MIN_VALUE`: a double just above the minimum value is used (*e.g.*, `1000.01`) 6. `ZERO_VALUE`: zero (`0`) is used Since `1000.00` is the minimum value, the data produced by the tests `1`, `2`, `3`, and `6` of the group `VALUE` are considered **invalid**, while `4` and `5` are not. For these tests considered **invalid**, the behavior defined in `Otherwise`, that is ``` Otherwise I must see "Salary must be greater than or equal to 1000" ``` is expected to happen. In other words, this behavior serves as [test oracle](https://en.wikipedia.org/wiki/Test_oracle) and must occur only when the produced value is invalid. Unlike this example, when the expected system behavior for invalid values is not specified and a test data is considered **invalid**, Concordia expects that test should **fail**. In this case, it generates the Test Case with the tag `@fail`. Now let's add **maximum value** restriction: ``` UI Element: Salary - data type is double - minimum value is 1000.00 Otherwise I must see "Salary must be greater than or equal to 1000" - maximum value is 30000.00 Otherwise I must see "Salary must be less than or equal to 30000" ``` All the tests of the group `RANGE` are now applicable. That is, the following tests will be included: 1. `MEDIAN_VALUE`: the median between the minimum and the maximum values 2. `RANDOM_BETWEEN_MIN_MAX_VALUES`: a pseudo-random double value between the minimum and the maximum values 3. `JUST_BELOW_MAX_VALUE`: the value just below the maximum value 4. `MAX_VALUE`: the maximum value 5. `JUST_ABOVE_MAX_VALUE`: the value just above the maximum value 6. `RANDOM_ABOVE_MAX_VALUE`: a pseudo-random double above the maximum value 7. `GREATEST_VALUE`: the greatest possible double The tests from `5` to `7` will produce values considered **invalid**. ### **Example 2** Let's define a user interface element named `Profession` and a table named `Professions` from which the values come from: ``` UI Element: Profession - type is select - value comes from "SELECT name from [Professions]" - required Table: Professions | name | | Lawyer | | Accountant | | Dentist | | Professor | | Mechanic | ``` Applicable test are: * `FILLED` * `NOT_FILLED` * `FIRST_ELEMENT` * `RANDOM_ELEMENT` * `LAST_ELEMENT` * `NOT_IN_SET` The first two tests are in the group `REQUIRED`. Since we declared `Profession` as having a *required* value, the test `FILLED` is considered **valid** but `NOT_FILLED` is not. Therefore, it is important to remember declaring required inputs accordingly. The last four tests are in the group `SET`. Only the last one, `NOT_IN_SET`, will produce a value considered **invalid**. ### **Example 3** In this example, let's adjust the past two examples to make `Salary` rules *dynamic* and change according to the `Profession`. Firstly, we add two columns the the `Professions` table: ``` Table: professions | name | min_salary | max_salary | | Lawyer | 100000 | 900000 | | Accountant | 90000 | 800000 | | Dentist | 150000 | 900000 | | Professor | 80000 | 500000 | | Mechanic | 50000 | 120000 | ``` Then, we change the rules to retrieve the values from the table: ``` UI Element: Salary - data type is double - minimum value comes from the query "SELECT min_salary FROM [Professions] WHERE name = {Profession}" Otherwise I must see "The given Salary is less than the minimum value" - maximum value comes from the query "SELECT max_salary FROM [Professions] WHERE name = {Profession}" Otherwise I must see "The given Salary is greater than the maximum value" ``` The reference to the UI Element `{Profession}` inside the query, makes the rules of `Salary` depend on `Profession`. Every time a `Profession` is selected, the **minimum value** and the **maximum value** of `Salary` changes according to the columns `min_salary` and `max_salary` of the table `Professions`.