Creating a Plug-in

Introduction

Concordia Compiler uses plug-ins for generating test scripts, running them and reading their results.

A plug-in can...

• be coded in JavaScript or any language that "transpiles" to JavaScript, such as TypeScript (recommended), Dart, or CoffeeScript.

• generate code (test scripts) for any programming language or testing framework able to perform functional tests for web, desktop or mobile applications.

Requirements for a plug-in

1. Its name must start with concordialang-, for example: concordialang-my-awesome-plugin.

2. It must be installable with NPM. We recommend you to publish it at NPM too.

3. It must implement the interface Plugin from concordialang-plugin.

4. Its package.json file must contain the property concordiaPlugin, like in the following example:

{
   ...
   
   "concordiaPlugin": {
      "file": "dist/index.js",
      "class": "MyAwesomePlugin",
      "serve": "npx command-to-start-my-testing-server"
   }
}

Properties of concordiaPlugin:

• file: string: Relative path of the file that contains an implementation of the Plugin interface.

• class: string: Class (name) that implementsPlugin.

• serve: string: Multi-platform console command to execute in order to start the testing server. Whether the testing framework does not need running a testing server, you can set something like echo \"Running a testing server is not needed.\".

Tasks to perform

A plug-in must deal with three tasks:

1. Transforming Abstract Test Scripts into test scripts (i.e., source code).

2. Executing the produced test scripts, regarding some execution options.

3. Transforming test execution results (produced by the testing framework) to the expected resulting format.

Let's get into the details on how to accomplish them.

Transforming Abstract Test Scripts into test scripts

Think of an Abstract Test Script (ATS) as an object to transform into source code. Basically, it contains a structure that represents the actions to be converted into commands of your preferred programming language.

For instance, consider the following action:

{
    "action": "click",
    "targets": ["#ok"]
}

That could be converted to the following command in Codeception, for PHP:

I->click("#ok");

Or to the following command in Cypress, for JavaScript:

cy.get('#ok').click();

Let's consider a more realistic example, starting with a Test Case in the hypothetical file feature1.testcase:

import "feature1.feature"

@generated
@scenario(1)
@variant(1)
Test Case: Scenario 1 | V1
  Given that I am on "https://myapp.com/register"
  When I fill <#email> with "john.doe@example.com"
    and I fill <#nome> with "John Doe"
    and I click on <#register>
  Then I see "Welcome, John Doe"

That would probably produce an Abstract Test Script like this:

{
    "feature": { "name": "Feature 1" },
    "scenario": { "name": "Scenario 1" },
    "testcases": [
        {
            "scenario": "V1",
            "commands": [
                {
                    "action": "amOn",
                    "values": [ "https://myapp.com/register" ],
                    "location": { "line": 7, "column": 2 }
                },
                {
                    "action": "type",
                    "targets": [ "#email" ],
                    "values": [ "john.doe@example.com" ],
                    "location": { "line": 8, "column": 2 } 
                },            
                {
                    "action": "type",
                    "targets": [ "#name" ],
                    "values": [ "John Doe" ],
                    "location": { "line": 9, "column": 4 }                  
                },                
                {
                    "action": "click",
                    "targets": [ "#register" ],
                    "location": { "line": 10, "column": 4 } 
                },
                {
                    "action": "see",
                    "values": [ "Welcome, John Doe" ],
                    "location": { "line": 11, "column": 2 }
                }                                         
            ]
        }
    ]
}

A plug-in for Cypress could generate the following code from the above object:

context("Feature 1", function() {

    it("Scenario 1 | V1", function() {
        cy.visit("https://myapp.com/register"); // (7,2)
        cy.get("#email").type("john.doe@example.com"); // (8,2)
        cy.get("#name").type("John Doe"); // (9,4)
        cy.get("#register").click(); // (10,4)
        cy.get("body").should("contain", "Welcome, John Doe"); // (11,2)
    });

});

To help generating such commands, you can use some template library such as Mustache (recommended) or Handlebars.

You can observe that after every command there is a line comment containing the corresponding location in the .testcase file. These comments will help to track the specification in case of test script failures.

Executing the test scripts

To execute the produced test scripts, you can use the target framework API or run the tool. Your call.

For example, whether you chose to run Cypress through npx and generate a report using Mocha JUnit Reporter, the command to run could be the following:

npx cypress run --reporter junit

Transforming execution results

Now the plug-in needs to read the report and transform it into an object of the class TestScriptExecutionResult that will be returned to Concordia Compiler.

For instance, the core of the plug-in for CodeceptJS reads the report from a JSON file generated by using CodeceptJS with Mocha Multi Reporters. The current implementation is available in the class ReportConverter.

A plug-in must be able to track back the location of failures or errors in the corresponding .testcase file from a test script. When a test script has a failure or an error, the report saves the stack trace where it occurred. For instance, tests/feature1.js:20:4 indicates, respectively, the file, line and column where the problem happened. You can use these data to read the line of the test script file and retrieve the line comment that contains the corresponding line in the .testcase file.

You can use a regular expression to extract the data from the stack trace, like ReportConverter did in the method extractScriptLocationFromStackTrace. Then you can use the class FileInstrumentationReader from concordialang-plugin to retrieve the location from the line comment. Example:

const scriptLocation = /* extract from stack trace */;

// Suppose that `failureLocation` is:
// {
//    path: "tests/feature1.js",
//    line: 7,
//    column: 8
// }

// Retrieve the comment with the reader:
const specLocation = reader.retrieveSpecLocation(
    scriptLocation
    );
    
// `specLocation` would be:
// {
//    path: "features/feature1.testcase",
//    line: 10,
//    column: 4
// }

You can read these locations in the examples described earlier. Whether tests/feature1.js:7:8 is...

cy.get("#register").click(); // (10,4)

Retrieving the line 10 and column 4 fromfeature1.testcase would give us:

  and I click on <#register>

In TestScriptExecutionResult, both test failures and errors are represented as an exception which includes the properties scriptLocation and specLocation.

Basic guide

We recommend you to follow these steps when creating a plug-in:

1. Open an issue to indicate that you are interested in developing a new plug-in for a certain testing framework or programming language. That is good way to make others aware of your work.

2. Choose a good name for your plug-in and check whether it registered at NPM. Remember that it must start with concordialang-. We recommend that it includes the target framework, tools or language. Example: concordialang-selenium-python to create a plug-in that generates test scripts for the Selenium framework and the Python language. You may use the opened issue (step 1) to get feedback about the name, if you wish.

3. Create a repository for it (e.g., at GitHub, GitLab, BitBucket) and clone the repository.

4. Create a NPM package with npm --init.

5. Add the property concordiaPlugin to your package.json file.

6. Install concordialang-plugin as a dependency:

npm install --save concordialang-plugin

7. Add your files. We recommend the following structure, but you can adapt it as you wish. For example, if you use TypeScript, you will probably add a tsconfig.json file. If you use Jest, your test folder will probably be named __tests__. We also recommend you to add a .editorconfig file and a .gitignore file (the latter should include the folder node_modules).

 ┃
 ┣ src/            🡐 your source code
 ┣ test/           🡐 your tests
 ┣ .editorconfig
 ┣ .gitignore
 ┗ package.json

8. Implement the interface Plugin. We recommend you to take a look at other plugins that did a similar job and encourage you to unit test all your code.

9. Test it with the Concordia Compiler before publishing it. Create a simple project that uses your plug-in. You can use NPM to install your plug-in from a directory in your computer or your remote repository. For example:

cd path/to/your/test-project/
npm install /path/to/your-plugin

10. Publish your plug-in at NPM after making sure that you are using Semantic Versioning. Congratulations! Tell everybody and ask for feedback! ✌

npm publish