Cucumber

AIO Tests provides integration with Cucumber tests by allowing users to import their Cucumber-based tests/scenarios into AIO Tests, either by mapping to existing cases or by creating new ones from the Cucumber results JSON file. Since AIO Tests rely only on the JSON report generated by Cucumber, the tests can be written in any of the supporting languages like Ruby, Perl, Java, Javascript, etc.

Having the automated results reporting in AIO Tests can help in having a holistic view of the coverage of requirements (unit to manual to functional and end-to-end automated tests).

This guide discusses how to import your existing Cucumber scenarios as AIO Tests and how to capture their execution results in AIO.

In this documentation, you’ll understand:

1 Cucumber
- 1.1 Sample Cucumber Scenario
- 1.2 Generating a JSON Report
- 1.3 Cucumber Sample JSON Report
2 Status Mapping Cucumber → AIO Tests
3 Mapping Cucumber Tests to AIO Tests
4 Mapping a Case to a Folder
5 Scenario Outline
- 5.1 Cases created before Scenario Outline/Examples support in AIO Tests
- 5.2 Cases created after Scenario Outline/Examples support in AIO Tests
6 Screenshots for Failed Cases
7 Importing Results

Cucumber

Cucumber is an open-source framework facilitating Behavior Driven Development (BDD), which helps create a formal understanding of system behavior among all stakeholders, from the business to the technical teams.

Cucumber is driven by simple domain-specific language (DSL) called Gherkin, which uses natural-language constructs like Given-When-Then to describe a system’s behavior and its expected outcomes.

Sample Cucumber Scenario

Gherkin uses a set of keywords to formalize a behavior. Most lines start with one of the keywords.

Cucumber tests can be structured as Features. A feature can look at encapsulating a particular functionality of a system.

Features are made of Scenarios, which are essentially different user flows of the functionality being tested.

Scenarios are made of Steps. Each step starts with Given, When, Then or And.

Feature: Capture the number of trees planted and show relation to global warming

  @P0 
  Scenario: Trees effect on global warming
    Given There are no trees numbered in a given region
    When Each user plants a tree and updates
    Then The temperature is updated in the temperature log
    When Temperature is captured over a few days
    Then The graph should show the relation between the number of trees and temperature

Generating a JSON Report

Cucumber supports generating JSON reports by default, amongst other formats like HTML, JUnit etc.

For e.g., to generate a JSON report in Java, either in JUnit or TestNG, use the annotation @CucumberOptions

import io.cucumber.testng.AbstractTestNGCucumberTests;
import io.cucumber.testng.CucumberOptions;

@CucumberOptions(
        features = "src/test/resources/features",
        glue = {"com.aiotests.samples.stepdefs"},
        plugin = { "pretty", "json:target/cucumber-reports/Cucumber.json",
                "junit:target/cucumber-reports/Cucumber.xml"})

public class MainRunner extends AbstractTestNGCucumberTests {
}

@RunWith(Cucumber.class)
@CucumberOptions(
        features = "src/test/resources/functionalTests",
        glue= {"stepDefinitions"},
        plugin = { "pretty" },
        monochrome = true
)
public class JUnitRunner {
}

Cucumber Sample JSON Report

The JSON report generated by Cucumber gives a detailed step-by-step report as follows :

Status Mapping Cucumber → AIO Tests

Cucumber JSON	Description	AIO Tests Mapping	Run Status

Cucumber JSON	Description	AIO Tests Mapping	Run Status
Passed	Passed step	Passed	Passed, if and only if, all steps are passed
Failed	Failed step	Failed	Failed, if even a single step is failed
Skipped	Step was not executed due to failure in a previous step	Not Run	In Progress, if one step is Not Run and other steps are passed. Failed , if any step is failed and others are Not Run
Undefined	Step does not have a corresponding implemented step definition	Not Run	In Progress, if one step is Not Run and other steps are passed. Failed , if any step is failed and others are Not Run

Mapping Cucumber Tests to AIO Tests

Scenario 1: I have existing automation scenarios, but do not have any cases corresponding to them in AIO Tests. I want to create cases from my existing automation tests.

Since these tests do not exist in AIO Tests yet, you can simply Import the Cucumber JSON report, either through the Import Results REST API or through the Cycles tab in AIO Tests.

Make sure to check the option to create new cases if no mapping is found option.

On import, cases would be created with the following mapping from the JSON report

Each scenario becomes one test case in AIO and a unique case key would be generated eg. AT-TC-1211.

The automation key [“id” from JSON report], becomes the map between a scenario and a case in AIO Tests.

If your scenario name changes then the id in Cucumber report changes as well.
In case, this changes often in your case, it is advisable to map the key of the case generated, as illustrated below, for future updations.

Scenario 2: I have created my scenarios in AIO Tests. Now I am automating those. How do I define a mapping?

To define a mapping between an existing case in AIO Tests and an automated case, simply add the case key tag [@AT-TC-1222] to the scenario as below:

Scenario 3: I have mapped scenarios in my feature files to AIO Tests. My steps have been updated. Do I need to update them in AIO Tests as well?

Once a mapping has been established between an AIO Test case and an automated scenario, the feature files can become the source of truth for future updates by force updating results while importing results.

If results are being imported from the Cycle Details window, the Update cases with scenario steps option update the cases in AIO Tests with those from the automated results.

Mapping a Case to a Folder

Users have the option to designate a default folder for newly created cases. They can also define a mapping between a case and a folder by simply adding a tag starting with the prefix @AIO-FOLDER- and mentioning the folder hierarchy. Folder hierarchy can be separated using / , for example - @AIO-FOLDER-Reports/Audit log.

This functionality is applicable only:

If a new case is generated through the import process
If the folder structure specified in the tag already exists.
Additionally, if configured, this will supersede the default folder setting.

Mapping Using the Default Folder Option

Mapping Each Case Using Tags

Scenario Outline

Cases created before Scenario Outline/Examples support in AIO Tests

Before the Data-driven feature was released, each example in a scenario outline was used to create a new case on the import of results.
Cases which have been created in this manner will continue to work and be updated as individual cases. They can be identified by the existing automation keys or by case keys if they were mapped at the Example level.
These cases can continue to be force-updated from the results file too.

If users wish to continue to work in this way, users can select to check “Consider outline examples as separate cases”. This would continue to create examples as individual cases and not use Example-based AIO cases.

Cases created after Scenario Outline/Examples support in AIO Tests

Dataset-based cases will be reported as single cases but with each example case being considered as a detail under the parent case as below:

Each example will have its own execution data and the status will be percolated up to the parent case status.

Post the support of DataSets/Examples/Scenario Outline in AIO Tests, when importing results for Scenario Outline, it is not possible to create cases from the results file, since the results file does not provide appropriate information regarding parameter names. Due to this limitation, the scenario outline cases can neither be created nor be force updated.
Such cases will be errored out with the following error:
Scenario Outline case doesn't exist and cannot be created from the results file. Please create or import the dataset-based case and use the case key to identify the scenario outline

It is mandatory that cases exist in the AIO Tests Case list and they are mapped using the case keys. This can be done by using the Import Feature Files feature. Feature files can also be used now to update scenarios using the Import Feature Files functionality.

Identifying dataset to update

Steps will be matched with replaced data parameter values and if matched, will be updated else an error will be thrown.
For cases not yet added to the cycle, steps will be matched with replaced data parameter steps. If the steps of the incoming scenario (example-based), match with a single dataset, the whole case will be added and all datasets will be created. Updation would be done of only those which are matching the values.
For cases in the cycle, if create new run is selected, steps will be matched against the last version of the case and updated if they match. If some of the dataset runs have been deleted, it will not impact the creation of new runs.
For cases in the cycle, if creating a new run is false, if some dataset runs have been deleted, then such scenarios should error out.
The order of datasets doesn’t matter.

Overall status and effort

The overall status of the encapsulating run would be updated based on the same rules that guide the update of the parent run when being interacted from the UI.
An effort will be added for each dataset-based run and updated in the parent run.

Screenshots for Failed Cases

Cucumber allows screenshots to be embedded in the JSON report, either in an @AfterStep annotation or @After annotation.

On importing the Cucumber results JSON, AIO Tests imports these embedded screenshots and attaches them to the runs.

AIO Tests imports screenshots only for failed cases and steps.
Embeddings captured @AfterStep, which executes at the end of each step, are attached as a step attachment if the step has failed.
Embeddings captured @After, which executes at the end of the scenario, are attached as a run attachment if the scenario has failed.
It is mandatory to set the filename and extension along with the mime type as shown below:

Importing Results

Post execution of a Cucumber suite, the cucumber-<xxx>.json file can be uploaded either via

AIO Tests REST APIs call using multipart form data to upload file
Cycle options in the Cycles Tab inside AIO Tests

Please follow the above links to continue to import results using either of the options.

For further queries and suggestions, please feel free to reach out to our customer support service via help@aiotests.com.