✔️
Concordia
  • Quick Start
  • Introduction
    • Understanding Concordia
    • Overview of the Language
    • A Small Example
    • Using a Plug-in
    • Using Databases
    • Command Line Interface
    • Configuration file
    • Related things
  • Language
    • Concordia Language
    • Actions
  • How It Works
    • Overview
    • Test Coverage and Techniques
  • Upgrade
    • How to Upgrade
    • Migration Guide
    • Breaking Changes
    • What's Next
  • Development
    • Creating a Plug-in
    • Actions vs APIs
    • Language Additions
Powered by GitBook
On this page
  • Introduction
  • Requirements for a plug-in
  • Tasks to perform
  • Transforming Abstract Test Scripts into test scripts
  • Executing the test scripts
  • Transforming execution results
  • Basic guide
Export as PDF
  1. Development

Creating a Plug-in

PreviousWhat's NextNextActions vs APIs

Last updated 4 years ago

Introduction

Concordia Compiler uses plug-ins for generating test scripts, running them and reading their results.

A plug-in can...

  • be coded in JavaScript or , such as (recommended), , or .

  • generate code (test scripts) for any programming language or testing framework able to perform for web, desktop or mobile applications.

Requirements for a plug-in

  1. Its name must start with concordialang-, for example: concordialang-my-awesome-plugin.

  2. It must be installable with . We recommend you to publish it at NPM too.

  3. It must implement the interface from .

  4. Its package.json file must contain the property concordiaPlugin, like in the following example:

{
   ...
   
   "concordiaPlugin": {
      "file": "dist/index.js",
      "class": "MyAwesomePlugin",
      "serve": "npx command-to-start-my-testing-server"
   }
}

Properties of concordiaPlugin:

  • file: string: Relative path of the file that contains an implementation of the Plugin interface.

  • class: string: Class (name) that implementsPlugin.

  • serve: string: Multi-platform console command to execute in order to start the testing server. Whether the testing framework does not need running a testing server, you can set something like echo \"Running a testing server is not needed.\".

Tasks to perform

A plug-in must deal with three tasks:

Let's get into the details on how to accomplish them.

Transforming Abstract Test Scripts into test scripts

Think of an Abstract Test Script (ATS) as an object to transform into source code. Basically, it contains a structure that represents the actions to be converted into commands of your preferred programming language.

For instance, consider the following action:

{
    "action": "click",
    "targets": ["#ok"]
}
I->click("#ok");
cy.get('#ok').click();

Let's consider a more realistic example, starting with a Test Case in the hypothetical file feature1.testcase:

import "feature1.feature"

@generated
@scenario(1)
@variant(1)
Test Case: Scenario 1 | V1
  Given that I am on "https://myapp.com/register"
  When I fill <#email> with "john.doe@example.com"
    and I fill <#nome> with "John Doe"
    and I click on <#register>
  Then I see "Welcome, John Doe"

That would probably produce an Abstract Test Script like this:

{
    "feature": { "name": "Feature 1" },
    "scenario": { "name": "Scenario 1" },
    "testcases": [
        {
            "scenario": "V1",
            "commands": [
                {
                    "action": "amOn",
                    "values": [ "https://myapp.com/register" ],
                    "location": { "line": 7, "column": 2 }
                },
                {
                    "action": "type",
                    "targets": [ "#email" ],
                    "values": [ "john.doe@example.com" ],
                    "location": { "line": 8, "column": 2 } 
                },            
                {
                    "action": "type",
                    "targets": [ "#name" ],
                    "values": [ "John Doe" ],
                    "location": { "line": 9, "column": 4 }                  
                },                
                {
                    "action": "click",
                    "targets": [ "#register" ],
                    "location": { "line": 10, "column": 4 } 
                },
                {
                    "action": "see",
                    "values": [ "Welcome, John Doe" ],
                    "location": { "line": 11, "column": 2 }
                }                                         
            ]
        }
    ]
}

A plug-in for Cypress could generate the following code from the above object:

context("Feature 1", function() {

    it("Scenario 1 | V1", function() {
        cy.visit("https://myapp.com/register"); // (7,2)
        cy.get("#email").type("john.doe@example.com"); // (8,2)
        cy.get("#name").type("John Doe"); // (9,4)
        cy.get("#register").click(); // (10,4)
        cy.get("body").should("contain", "Welcome, John Doe"); // (11,2)
    });

});

You can observe that after every command there is a line comment containing the corresponding location in the .testcase file. These comments will help to track the specification in case of test script failures.

Executing the test scripts

To execute the produced test scripts, you can use the target framework API or run the tool. Your call.

npx cypress run --reporter junit

Transforming execution results

A plug-in must be able to track back the location of failures or errors in the corresponding .testcase file from a test script. When a test script has a failure or an error, the report saves the stack trace where it occurred. For instance, tests/feature1.js:20:4 indicates, respectively, the file, line and column where the problem happened. You can use these data to read the line of the test script file and retrieve the line comment that contains the corresponding line in the .testcase file.

const scriptLocation = /* extract from stack trace */;

// Suppose that `failureLocation` is:
// {
//    path: "tests/feature1.js",
//    line: 7,
//    column: 8
// }

// Retrieve the comment with the reader:
const specLocation = reader.retrieveSpecLocation(
    scriptLocation
    );
    
// `specLocation` would be:
// {
//    path: "features/feature1.testcase",
//    line: 10,
//    column: 4
// }

You can read these locations in the examples described earlier. Whether tests/feature1.js:7:8 is...

cy.get("#register").click(); // (10,4)

Retrieving the line 10 and column 4 fromfeature1.testcase would give us:

  and I click on <#register>

Basic guide

We recommend you to follow these steps when creating a plug-in:

3. Create a repository for it (e.g., at GitHub, GitLab, BitBucket) and clone the repository.

4. Create a NPM package with npm --init.

5. Add the property concordiaPlugin to your package.json file.

npm install --save concordialang-plugin
 ┃
 ┣ src/            🡐 your source code
 ┣ test/           🡐 your tests
 ┣ .editorconfig
 ┣ .gitignore
 ┗ package.json

9. Test it with the Concordia Compiler before publishing it. Create a simple project that uses your plug-in. You can use NPM to install your plug-in from a directory in your computer or your remote repository. For example:

cd path/to/your/test-project/
npm install /path/to/your-plugin
npm publish

Transforming into test scripts (i.e., source code).

Executing the produced test scripts, regarding some .

Transforming test execution results (produced by the testing framework) to .

That could be converted to the following command in , for PHP:

Or to the following command in , for JavaScript:

To help generating such commands, you can use some template library such as (recommended) or .

For example, whether you chose to run Cypress through npx and generate a report using the command to run could be the following:

Now the plug-in needs to read the report and transform it into an object of the class that will be returned to Concordia Compiler.

For instance, the reads the report from a JSON file generated by using with . The current implementation is available in the class .

You can use a regular expression to extract the data from the stack trace, like did in the method extractScriptLocationFromStackTrace. Then you can use the class from to retrieve the location from the line comment. Example:

In , both test failures and errors are represented as an exception which includes the properties scriptLocation and specLocation.

1. to indicate that you are interested in developing a new plug-in for a certain testing framework or programming language. That is good way to make others aware of your work.

2. Choose a good name for your plug-in and check whether it registered at . Remember that it must start with concordialang-. We recommend that it includes the target framework, tools or language. Example: concordialang-selenium-python to create a plug-in that generates test scripts for the Selenium framework and the Python language. You may use the opened issue (step 1) to get feedback about the name, if you wish.

6. Install as a dependency:

7. Add your files. We recommend the following structure, but you can adapt it as you wish. For example, if you use , you will probably add a tsconfig.json file. If you use , your test folder will probably be named __tests__. We also recommend you to add a file and a file (the latter should include the folder node_modules).

8. Implement the interface . We recommend you to and encourage you to unit test all your code.

10. Publish your plug-in at after making sure that you are using . Congratulations! Tell everybody and ask for feedback! ✌

any language that "transpiles" to JavaScript
TypeScript
Dart
CoffeeScript
functional tests
NPM
Plugin
concordialang-plugin
Abstract Test Scripts
execution options
the expected resulting format
Codeception
Cypress
Mustache
Handlebars
Mocha JUnit Reporter,
TestScriptExecutionResult
core of the plug-in for CodeceptJS
CodeceptJS
Mocha Multi Reporters
ReportConverter
ReportConverter
FileInstrumentationReader
concordialang-plugin
TestScriptExecutionResult
Open an issue
NPM
concordialang-plugin
TypeScript
Jest
.editorconfig
.gitignore
Plugin
take a look at other plugins that did a similar job
NPM
Semantic Versioning