AIO Tests supports importing Nightwatch test results through it’s support for JUnit reports/Cucumber reports or via the AIO Tests REST APIs, which can be invoked from the hooks available in Nightwatch.

Nightwatch.js is an integrated framework for performing automated end-to-end testing on web applications and websites, across all major browsers. It is written in Node.js and uses the WebDriver API to interact with various browsers.

This document provides an overview on :

Generating Cucumber reports with Nightwatch + Cucumber and uploading it in AIO Tests
Using AIO Tests REST APIs to report results and much more, using the Nightwatch framework hooks.

Nightwatch + Cucumber Setup

npm init nightwatch
npm i @cucumber/cucumber --save-dev

In order to use CucumberJS in Nightwatch you need to set the test_runner config property and set the type to cucumber. You will also need to set the path to where the feature files are located. Additionally, you can configure where the report files would be generated.

{
  test_runner: {
    // set cucumber as the runner
    type: 'cucumber',
    options: {
      feature_path: 'test/features',
      format: 'json:report/cucumber_report.json',
      auto_start_session: true,
      parallel: 2 
    }
  },
  src_folders: ['examples/cucumber-js/features/step_definitions']
}

Mapping and Running your tests

In AIO Tests, a unique key PROJKEY-TC-12 exists for each case. The unique key can be added to the scenario tags to report results against it in AIO Tests.

Below example maps a scenario to an AIO Case

//test/features/nightwatch.feature
Feature: Google Search
  Background: Background name
    Given I open the Rijksmuseum page
    And I dismiss the cookie dialog
    Then the title is "Rijksmuseum Amsterdam, home of the Dutch masters"

  @PROJKEY-TC-12  @Sanity
  Scenario: Searching the Rijksmuseum with operation
    Given I search "night watch"
    Then Body contains "Operation Night Watch"

  @PROJKEY-TC-13 @P0
  Scenario: Searching the Rijksmuseum with night watch
    Given I search "night watch"
    Then Body contains "The Night Watch, Rembrandt van Rijn, 1642sakljdfl"

To trigger the Nightwatch cucumber tests:

npx nightwatch test/features/nightwatch

Reporting results via Cucumber.json

Nightwatch-Cucumber generates a cucumber.json report which can be directly imported in AIO Tests. All scenario that are not tagged with any case keys, will be created as new cases, along with the steps from the json report.

Nightwatch will forward --format and --format-options CLI arguments, if present, to Cucumber. Based on the configuration in nightwatch.conf.js, the cucumber.json report would be created at the path specified in the format field. In configuration above, the path was

      format: 'json:report/cucumber_report.json',

AIO Tests provides a rich set of APIs for Execution Management, using which users can not only report execution status, but also add effort, actual results, comments, defects and attachments to runs as well as steps.

The basic sample below will show how Cucumber Hooks can leverage the AIO Tests REST APIs to report results. We would be using the AfterAll hook of cucumber to achieve this.

//aio-reporting.js
const {AfterAll} = require('@cucumber/cucumber');
const fs = require("fs");
const FormData = require('form-data');
const axios = require('axios');

const AIO_API_BASEURL = "https://tcms.aioreports.com/aio-tcms/api/v1/project"
const JIRA_PROJ = "NVTES"
const AIO_API_KEY = "yourapikey"

AfterAll(  function() {
    console.log("Reporting results to AIO Tests")
    //timeout is important - since the report gets generated in some time
    setTimeout(() => {
        let promises = [];
        fs.readdirSync("./report/").forEach(file => {
            console.log("Uploading file to AIO" + file)
            var bodyFormData = new FormData();
            bodyFormData.append('file', fs.createReadStream("./report/" + file));
            bodyFormData.append('createCase', 'true');
            bodyFormData.append('bddForceUpdateCase','true');
            promises.push(axios.post(`${AIO_API_BASEURL}/${JIRA_PROJ}/testcycle/NVTES-CY-666/import/results?type=Cucumber`,
                bodyFormData, {
                    headers: {
                        'Authorization': `AioAuth ${AIO_API_KEY}`,
                        'Content-Type': 'multipart/form-data'
                    }
                }).then(function (response) {
                console.log(response.data)
            }).catch((e) => console.log(e.response.data)))
        });
        return Promise.all(promises);
    }, 2000);
});

To trigger the Nightwatch tests with the above file, the reporting file needs to be provided to cucumber-js, using the following command:

npx nightwatch test/features/nightwatch --require aio_reporting.js

Reporting results for each scenario individually

Reporting scenario wise result can come handy in couple of scenarios:

No mapping is done using key and there is some other way to identify tests either via automation key or some custom fields
One to many mapping i.e. one automated test covers multiple manual cases.
There are long running suites and there is a need to see runtime results
BDD Cases map to classic cases in AIO Tests and step wise results are not required

For any of the above requirements, users can use the Mark TC Result API to report individual results.

The below sample demonstrates, where a scenario tag maps to one of the fields in AIO Tests, (automation key in sample). We would be using the After hook of cucumber to achieve this. The steps would be to

Find AIO cases based on search criteria
Report result against each case using bulk update request

//aio-reporting.js
const {After} = require('@cucumber/cucumber');
const axios = require('axios');

const AIO_API_BASEURL = "https://tcms.aioreports.com/aio-tcms/api/v1/project"
const JIRA_PROJ = "NVTES"
const AIO_API_KEY = "your key"

After(async function (scenario) {

    const aioCaseKeys = [];
    await fetchAIOCases(scenario.pickle.tags, aioCaseKeys)

    if (aioCaseKeys.length) {
        await markCaseResult(scenario, aioCaseKeys)
    }
    return;
})

async function fetchAIOCases(scenarioTags, aioCaseKeys) {
    for (const t of scenarioTags) {
        if (t.name.startsWith("@MGM-")) {
            let searchQuery = {
                "automationKey": {
                    "comparisonType": "EXACT_MATCH",
                    "value": t.name.substring(1)
                }
            };
            const caseSearchResponse = await axios({
                method: 'post',
                url: `${AIO_API_BASEURL}/${JIRA_PROJ}/testcase/search`,
                headers: {'Authorization': `AioAuth ${AIO_API_KEY}`},
                data: searchQuery
            })
            if (caseSearchResponse.data.items?.length) {
                caseSearchResponse.data.items.forEach(i => aioCaseKeys.push(i.key));
            }
        }
    }
}

async function markCaseResult(scenario, aioCaseKeys) {
    let bulkUpdateRequest = {"testRuns": []};
    for (const aioCaseKey of aioCaseKeys) {
        let caseStatus = {
            "testCaseKey": aioCaseKey,
            //Can be enhanced based on statuses
            "testRunStatus": scenario.result.status === 'PASSED' ? "Passed" : scenario.result.status === "FAILED" ? "Failed": "Not Run",
            //comments can be added
            "effort": scenario.result.duration.seconds
        }
        bulkUpdateRequest.testRuns.push(caseStatus);
    }
    try {
        const response = await axios({
            method: 'post',
            url: `${AIO_API_BASEURL}/${JIRA_PROJ}/testcycle/NVTES-CY-437/bulk/testrun/update`,
            headers: {'Authorization': `AioAuth ${AIO_API_KEY}`},
            data: bulkUpdateRequest
        })
    } catch (e) {
        console.log(e.response.data)
    }
}