Creating a Plug-in
Concordia Compiler uses plug-ins for generating test scripts, running them and reading their results.
A plug-in can...
- be coded in JavaScript or any language that "transpiles" to JavaScript, such as TypeScript (recommended), Dart, or CoffeeScript.
- generate code (test scripts) for any programming language or testing framework able to perform functional tests for web, desktop or mobile applications.
- 1.Its name must start with
concordialang-
, for example:concordialang-my-awesome-plugin
. - 2.
- 3.
- 4.Its
package.json
file must contain the propertyconcordiaPlugin
, like in the following example:
{
...
"concordiaPlugin": {
"file": "dist/index.js",
"class": "MyAwesomePlugin",
"serve": "npx command-to-start-my-testing-server"
}
}
Properties of
concordiaPlugin
:file: string
: Relative path of the file that contains an implementation of thePlugin
interface.class: string
: Class (name) that implementsPlugin
.serve: string
: Multi-platform console command to execute in order to start the testing server. Whether the testing framework does not need running a testing server, you can set something likeecho \"Running a testing server is not needed.\"
.
A plug-in must deal with three tasks:
- 1.
- 2.
- 3.Transforming test execution results (produced by the testing framework) to the expected resulting format.
Let's get into the details on how to accomplish them.
Think of an Abstract Test Script (ATS) as an object to transform into source code. Basically, it contains a structure that represents the actions to be converted into commands of your preferred programming language.
For instance, consider the following action:
{
"action": "click",
"targets": ["#ok"]
}
I->click("#ok");
cy.get('#ok').click();
Let's consider a more realistic example, starting with a Test Case in the hypothetical file
feature1.testcase
:import "feature1.feature"
@generated
@scenario(1)
@variant(1)
Test Case: Scenario 1 | V1
Given that I am on "https://myapp.com/register"
When I fill <#email> with "[email protected]"
and I fill <#nome> with "John Doe"
and I click on <#register>
Then I see "Welcome, John Doe"
That would probably produce an Abstract Test Script like this:
{
"feature": { "name": "Feature 1" },
"scenario": { "name": "Scenario 1" },
"testcases": [
{
"scenario": "V1",
"commands": [
{
"action": "amOn",
"values": [ "https://myapp.com/register" ],
"location": { "line": 7, "column": 2 }
},
{
"action": "type",
"targets": [ "#email" ],
"values": [ "[email protected]" ],
"location": { "line": 8, "column": 2 }
},
{
"action": "type",
"targets": [ "#name" ],
"values": [ "John Doe" ],
"location": { "line": 9, "column": 4 }
},
{
"action": "click",
"targets": [ "#register" ],
"location": { "line": 10, "column": 4 }
},
{
"action": "see",
"values": [ "Welcome, John Doe" ],
"location": { "line": 11, "column": 2 }
}
]
}
]
}
A plug-in for Cypress could generate the following code from the above object:
context("Feature 1", function() {
it("Scenario 1 | V1", function() {
cy.visit("https://myapp.com/register"); // (7,2)
cy.get("#email").type("[email protected]"); // (8,2)
cy.get("#name").type("John Doe"); // (9,4)
cy.get("#register").click(); // (10,4)
cy.get("body").should("contain", "Welcome, John Doe"); // (11,2)
});
});
To help generating such commands, you can use some template library such as Mustache (recommended) or Handlebars.
You can observe that after every command there is a line comment containing the corresponding location in the
.testcase
file. These comments will help to track the specification in case of test script failures.To execute the produced test scripts, you can use the target framework API or run the tool. Your call.
For example, whether you chose to run Cypress through
npx
and generate a report using Mocha JUnit Reporter, the command to run could be the following:npx cypress run --reporter junit
Now the plug-in needs to read the report and transform it into an object of the class
TestScriptExecutionResult
that will be returned to Concordia Compiler.For instance, the core of the plug-in for CodeceptJS reads the report from a JSON file generated by using CodeceptJS with Mocha Multi Reporters. The current implementation is available in the class
ReportConverter
.A plug-in must be able to track back the location of failures or errors in the corresponding
.testcase
file from a test script. When a test script has a failure or an error, the report saves the stack trace where it occurred. For instance, tests/feature1.js:20:4
indicates, respectively, the file, line and column where the problem happened. You can use these data to read the line of the test script file and retrieve the line comment that contains the corresponding line in the .testcase
file. You can use a regular expression to extract the data from the stack trace, like
ReportConverter
did in the method extractScriptLocationFromStackTrace
. Then you can use the class FileInstrumentationReader
from concordialang-plugin to retrieve the location from the line comment. Example:const scriptLocation = /* extract from stack trace */;
// Suppose that `failureLocation` is:
// {
// path: "tests/feature1.js",
// line: 7,
// column: 8
// }
// Retrieve the comment with the reader:
const specLocation = reader.retrieveSpecLocation(
scriptLocation
);
// `specLocation` would be:
// {
// path: "features/feature1.testcase",
// line: 10,
// column: 4
// }
You can read these locations in the examples described earlier. Whether
tests/feature1.js:7:8
is... cy.get("#register").click(); // (10,4)
Retrieving the line
10
and column 4
fromfeature1.testcase
would give us: and I click on <#register>
In
TestScriptExecutionResult
, both test failures and errors are represented as an exception
which includes the properties scriptLocation
and specLocation
.We recommend you to follow these steps when creating a plug-in:
1. Open an issue to indicate that you are interested in developing a new plug-in for a certain testing framework or programming language. That is good way to make others aware of your work.
2. Choose a good name for your plug-in and check whether it registered at NPM. Remember that it must start with
concordialang-
. We recommend that it includes the target framework, tools or language. Example: concordialang-selenium-python
to create a plug-in that generates test scripts for the Selenium framework and the Python language. You may use the opened issue (step 1) to get feedback about the name, if you wish.3. Create a repository for it (e.g., at GitHub, GitLab, BitBucket) and clone the repository.
4. Create a NPM package with
npm --init
.5. Add the property
concordiaPlugin
to your package.json
file.npm install --save concordialang-plugin
7. Add your files. We recommend the following structure, but you can adapt it as you wish. For example, if you use TypeScript, you will probably add a
tsconfig.json
file. If you use Jest, your test folder will probably be named __tests__
. We also recommend you to add a .editorconfig
file and a .gitignore
file (the latter should include the folder node_modules
). ┃
┣ src/ 🡐 your source code
┣ test/ 🡐 your tests
┣ .editorconfig
┣ .gitignore
┗ package.json
8. Implement the interface Plugin. We recommend you to take a look at other plugins that did a similar job and encourage you to unit test all your code.
9. Test it with the Concordia Compiler before publishing it. Create a simple project that uses your plug-in. You can use NPM to install your plug-in from a directory in your computer or your remote repository. For example:
cd path/to/your/test-project/
npm install /path/to/your-plugin
10. Publish your plug-in at NPM after making sure that you are using Semantic Versioning. Congratulations! Tell everybody and ask for feedback! ✌
npm publish
Last modified 2yr ago