Our documentation site has been updated to AIO Tests Knowledge Base

Cucumber

Owned by Noopur Varshney [AIO Tests]
Last updated: Feb 14, 2024 by Samyak Awale
9 min read

AIO Tests provides integration with Cucumber tests by allowing users to import their Cucumber based tests/scenarios into AIO Tests, either by mapping to existing cases or by creating new ones from the Cucumber results json file. Since AIO Tests relies only on the JSON report generated by Cucumber, the tests can be written in any of the supporting languages like Ruby, Perl, Java, Javascript, etc.

Having the automated results reporting in AIO Tests, can help in having a holistic view of coverage of requirements (unit to manual to functional and end-to-end automated tests).

This article discusses how to import your existing Cucumber scenarios as AIO Tests and how to capture their execution results in AIO.

Cucumber

Cucumber is an open source framework facilitating Behavior Driven Development (BDD), which helps create a formal understanding of system behavior between all stakeholders, from the business to the technical teams.

Cucumber is driven by simple domain-specific language (DSL) called Gherkin, which uses natural-language constructs like Given-When-Then to describe a system’s behavior and its expected outcomes.

Sample Cucumber Scenario

Gherkin uses a set of keywords to formalize a behavior. Most lines start with one of the keywords.

Cucumber tests can be structured as Features. A feature can look at encapsulating a particular functionality of a system.

Features are made of Scenarios, which are essentially different user flows of the functionality being tested.

Scenarios are made of Steps. Each step starts with Given, When, Then or And.

Feature: Capture the number of trees planted and show relation to global warming @P0 Scenario: Trees effect on global warming Given There are no trees numbered in a given region When Each user plants a tree and updates Then The temperature is updated in the temperature log When Temperature is captured over a few days Then The graph should show the relation between the number of trees and temperature

Generating a JSON report

Cucumber supports generating JSON report by default, amongst other formats like HTML, JUnit etc.

For e.g., to generate a JSON report in Java, either in JUnit or TestNG, use the annotation @CucumberOptions

import io.cucumber.testng.AbstractTestNGCucumberTests; import io.cucumber.testng.CucumberOptions; @CucumberOptions( features = "src/test/resources/features", glue = {"com.aiotests.samples.stepdefs"}, plugin = { "pretty", "json:target/cucumber-reports/Cucumber.json", "junit:target/cucumber-reports/Cucumber.xml"}) public class MainRunner extends AbstractTestNGCucumberTests { }
@RunWith(Cucumber.class) @CucumberOptions( features = "src/test/resources/functionalTests", glue= {"stepDefinitions"}, plugin = { "pretty" }, monochrome = true ) public class JUnitRunner { }

Cucumber Sample JSON Report

The JSON report generated by Cucumber gives a detailed step by step report as follows :

Status Mapping Cucumber → AIO Tests

Cucumber JSON

Description

AIO Tests Mapping

Run Status

Cucumber JSON

Description

AIO Tests Mapping

Run Status

Passed

Passed step

Passed

Passed, if and only if, all steps are passed

Failed

Failed step

Failed

Failed, if even a single step is failed

Skipped

Step was not executed due to failure in a previous step

Not Run

In Progress, if one step is Not Run and other steps are passed.

Failed , if any step is failed and others are Not Run

Undefined

Step does not have a corresponding implemented step definition

Not Run

In Progress, if one step is Not Run and other steps are passed.

Failed , if any step is failed and others are Not Run

Mapping Cucumber tests to AIO Tests

Scenario 1 : I have existing automation scenarios, but do not have any cases corresponding to them in AIO Tests. I want to create cases from my existing automation tests.

Since these tests do not exist in AIO Tests yet, you can simply Import the Cucumber JSON report, either through the Import Results REST API or through the Cycles tab in AIO Tests.

Make sure to check the option to create new cases if no mapping is found option.

On import, cases would be created with the following mapping from JSON report

Each scenario becomes one testcase in AIO and a unique case key would be generated eg. AT-TC-1211.

The automation key [“id” from JSON report], becomes the map between a scenario and a case in AIO Tests.

Please note, if your scenario name changes then the id in Cucumber report changes as well.
In case, this changes often in your case, it is advisable to map the key of the case generated, as illustrated below, for future updations.

Scenario 2 : I have created my scenarios in AIO Tests. Now I am automating those. How do I define a mapping?

To define a mapping between an existing case in AIO Tests and an automated case, simply add the case key tag [@AT-TC-1222] to the scenario as below:

Scenario 3 : I have mapped scenarios in my feature files to AIO Tests. My steps have been updated. Do I need to update them in AIO Tests as well?

Once an mapping has been established between an AIO Test case and an automated scenario, the feature files can become the source of truth for future updates by force updating results while importing results.

If results are being imported from the Cycle Details window, the Update cases with scenario steps option, updates the cases in AIO Tests with those from the automated results.

Mapping a case to a folder

Users have the option to designate a default folder for newly created cases. They can also define a mapping between a case and a folder by simply adding a tag starting with the prefix @AIO-FOLDER- and mentioning the folder hierarchy. Folder hierarchy can be separated using / , for example - @AIO-FOLDER-Reports/Audit log
Note- This functionality is applicable only:

  • if a new case is generated through the import process

  • if the folder structure specified in the tag already exists.

    Additionally, if configured, this will supersede the default folder setting.

 

  1. Mapping using the default folder option

 

  1. Mapping each case using tags

 

Scenario Outline

Cases created before Scenario Outline/Examples support in AIO Tests

Before the Data driven feature was released, each example in a scenario outline used to create a new case on import of results.
Cases which have been created in this manner, will continue to work and be updated as individual cases. They can be identified by the existing automation keys or by case keys if they were mapped at Example level.
These cases can continue to be force updated from the results file too.

If users wish to continue to work in this way, user’s can select to check “Consider outline examples as separate cases”. This would continue to create examples as individual cases and not use Example based AIO cases.

Cases created after Scenario Outline/Examples support in AIO Tests

Dataset based cases will be reported as single cases but with each example case being considered as a detail under the parent case as below:

Each example will have it’s own execution data and the status will be percolated up to the parent case status.

Post the support of DataSets/Examples/Scenario Outline in AIO Tests, when importing results for Scenario Outline, it is not possible to create cases from the results file, since the results file does not provide appropriate information regarding parameter names. Due to this limitation, the scenario outline cases can neither be created nor be force updated.
Such cases will be errored out with the following error:
Scenario Outline case doesn't exist and cannot be created from results file. Please create or import Dataset based case and use casekey to identify scenario outline

It is mandatory that cases exist in the AIO Tests Case list and they are mapped using the case keys. This can be done by using the Import Feature Files feature. Feature files can also be used now to update scenarios using the Import Feature Files functionality.

Identifying dataset to update

  1. Steps will be matched with replaced data parameter values and if matched, will be updated else an error will be thrown.

  2. For cases not yet added to cycle, steps will be matched with replaced data parameter steps. If steps of incoming scenario (example based), match with a single dataset, the whole case would be added and all datasets will be created. Updation would be done of only those which are matching the values.

  3. For cases in the cycle, if create new run is selected, steps will be matched against the last version of the case and updated if they match. If some of the dataset runs have been deleted, it will not impact the creation of new runs.

  4. For cases in cycle, if create new run is false, if some dataset runs have been deleted, then such scenarios should error out

  5. Order of datasets doesn’t matter

Overall status and effort

  1. Overall status of encapsulating run would be updated based on same rules that guide the update of parent run when being interacted from the UI.

  2. Effort will be added for each dataset based run and updated in parent run.

Screenshots for failed Cases

Cucumber allows screenshots to be embedded in the JSON report, either in an @AfterStep annotation or @After annotation.

On importing the Cucumber results JSON, AIO Tests imports these embedded screenshots and attaches it to the runs.

  • AIO Tests imports screenshots only for failed cases and steps.

  • Embeddings captured @AfterStep, which executes at the end of each step, is attached as a step attachment, if the step has failed.

  • Embeddings captured @After, which executes at the end of scenario, is attached as a run attachment, if the scenario has failed.

  • It is mandatory to set filename and extension along with mime type as shown below:

Importing Results

Post execution of a Cucumber suite, the cucumber-<xxx>.json file can be uploaded either via

Please follow the above links to continue to import results using either of the options.