Versions Compared

Old Version 1

changes.mady.by.user Noopur Varshney [AIO Tests]

Saved on

compared with

New Version Current

changes.mady.by.user Noopur Varshney [AIO Tests]

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • Generating the Junit report from Playwright tests and uploading it to AIO Tests.

  • Generating Cucumber reports with Playwright + Cucumber and uploading it in AIO Tests

  • Using AIO Tests REST APIs to report results and much more, using the Playwright framework hooks.

Table of Contents

Playwright + JUnit

The demo example is based on the Getting Started project of Playwright.

Required Playwright Setup

  1. npm init playwright@latest. On prompt, select the tests folder

  2. For reporting results

    1. For JUnit report : PLAYWRIGHT_JUNIT_OUTPUT_NAME=results.xml npx playwright test --reporter=junit

    2. For reporting results via AIO Tests API : npm install axios (This can be replaced with any library to make API calls)

Mapping and Running your tests

The sample test generated by Playwright is as follows.

...

To trigger the Playwright tests, use:

npx playwright test

Reporting results via JUnit file

Using the native JUnit reporter

Playwright Test comes with a few built-in reporters for different needs and ability to provide custom reporters. The easiest way to try out built-in reporters is to pass --reporter.

...

Expand
titlePlaywright JUnit xml
Code Block
<testsuites id="" name="" tests="1" failures="0" skipped="0" errors="0" time="4.201744999885559">
<testsuite name="example.spec.js" timestamp="1675073959913" hostname="" 
tests="1" failures="0" skipped="0" time="2.681" errors="0">
<testcase name="New Todo PROJ1-TC-23 : should allow me to add todo items" 
classname="[chromium] › example.spec.js › New Todo › PROJ1-TC-23 : should allow me to add todo items" time="2.681">
</testcase>
</testsuite>
</testsuites>

Uploading results to AIO Tests

Post execution of a suite, the TEST-<xxx>.xml file can be uploaded either via

...

If the same file is uploaded again, the cases will be identified using the automation key (classname.name )and would be updated, instead of creating new cases.

Status Mapping JUnit → AIO Tests

JUnit XML

Description

AIO Tests Mapping

No tag inside <testcase> means Passed

Passed case

Passed

</skipped>

Skipped case either by @Ignore or others

Not Run

</failure>

Indicates that the test failed. A failure is a test which the code has explicitly failed by using the mechanisms for that purpose. e.g., via an assertEquals.

Failed

</error>

Indicates that the test errored. An errored test is one that had an unanticipated problem. e.g., an unchecked throwable;

Failed

Reporting results via Playwright Hooks and AIO Tests REST APIs

AIO Tests provides a rich set of APIs for Execution Management, using which users can not only report execution status, but also add effort, actual results, comments, defects and attachments to runs as well as steps.
AIO Tests also provides APIs to create cycles and to add cases to cycles for execution planning.

The basic sample below will show how Mocha Hooks in WebdriverIO can leverage the AIO Tests REST APIs to report results. In the wdio.conf.js, the afterTest method can be used to make AIO API call.

Mocha aftestTest method

The afterTest call is defined as below: [ref https://webdriver.io/docs/configurationfile ]

Code Block
/**
     * Function to be executed after a test (in Mocha/Jasmine only)
     * @param {Object}  test             test object
     * @param {Object}  context          scope object the test was executed with
     * @param {Error}   result.error     error object in case the test fails, otherwise `undefined`
     * @param {Any}     result.result    return object of test function
     * @param {Number}  result.duration  duration of test
     * @param {Boolean} result.passed    true if test has passed, otherwise false
     * @param {Object}  result.retries   informations to spec related retries, e.g. `{ attempts: 0, limit: 0 }`
     */
    afterTest: function (test, context, { error, result, duration, passed, retries }) {
    },


Establish a convention for AIO Tests Case keys

For the purpose of the example, we have established a convention to map cases - the AIO Tests case key is the prefix to the case title ege.g. it('NVPROJ-TC-11: should login with valid credentials' contains NVPROJ-TC-11, which is the AIO Tests Case key.

...

Code Block
describe('My Login application', () => {
    it('NVPROJ-TC-11: should login with valid credentials', async () => {
        await LoginPage.open();

        await LoginPage.login('tomsmith', 'SuperSecretPassword!');
        await expect(SecurePage.flashAlert).toBeExisting();
        await expect(SecurePage.flashAlert).toHaveTextContaining(
            'You logged into a secure area!');
    });
    it('NVPROJ-TC-12: should login with invalid credentials : ', async () => {
        await LoginPage.open();

        await LoginPage.login('tomsmith', 'SuperSecretPassword1');
        await expect(SecurePage.flashAlert).toBeExisting();
        await expect(SecurePage.flashAlert).toHaveTextContaining(
            'You logged into a secure area!');
    });
});


Reporting result via API

Playwright provides a way to develop custom reporters.

...

Code Block
languagejs
const axios = require('axios');

class AIOReporter {

    promises = [];
    onTestEnd(test, result) {
        this.promises.push(this.postResults(test, result));
    }

    postResults(test, result) {
        if(test.title.startsWith("NVTES-TC-")) {
            let tc = test.title.split(":")[0].trim()
            console.log("Reporting results ..." + tc)
            let resultData = {"testRunStatus": result.status === 'passed'? "Passed" :"Failed", "effort": result.duration/1000};
            if(result.status !== 'passed')
            {
                resultData["comments"] = [result.error];
            }
            return axios({
                method: 'post',
                url: `https://tcms.aioreportsaiojiraaaps.com/aio-tcms/api/v1/project/NVTES/testcycle/NVTES-CY-434/testcase/${tc}/testrun`,
                headers: {'Authorization': 'AioAuth yourtokenhere'},
                data :resultData
            })
                .then(function (response) {
                    console.log(response.data)
                }).catch((e) => console.log(e.response.data));
        }
    }
    
    //On End returns a promise and hence it is important, since onTestEnd is not async
    onEnd(result) {
        return Promise.all(this.promises);
        //On end can also be used to simply import the Junit file in AIO, using the
        //Import results API.  For code sample of Import Results API, please refer 
        //to Reporting results via Cucumber Hooks and AIO Tests REST APIs sample
    }
}

module.exports = AIOReporter;

...

  1. It uses the test title to identify the case key [ based on the convention established]

  2. Create a POST request

    1. URL : For cloud the url host would be https://tcms.aioreportsaiojiraapps.com/aio-tcms/api/v1. For Jira server, it would be the native Jira server hostname.

    2. Authorization : Please refer to /wiki/spaces/ATDoc/pages/1499594753 Rest API Authentication to understand how to authorize users. The authentication information goes in the headers: {'Authorization': '<Auth based on Jira Cloud/Jira Server>'},

    3. POST Body : The body consists of data from the test and result object provided by Playwright. If the case has failed, the error is posted as comments.

    4. If required, the basic example can be extended to upload attachments against the case using the upload attachment API.

Info

The above is a basic example of what can be done with the hooks and AIO Tests APIs. It is recommended to add appropriate error handling and enhance it based on your automation requirements.

 

Playwright + Cucumber Setup

  1. npm init playwright@latest. On prompt, select the tests folder

  2. npm install @cucumber/cucumber.

Mapping and Running your tests

Cucumber generates a cucumber.json report which can be directly imported in AIO Tests.

...

Code Block
languagejson
[
  {
    "description": "",
    "elements": [
      {
        "description": "",
        "id": "minimal-scenario;justified-reasoning",
        "keyword": "Scenario",
        "line": 5,
        "name": "Justified reasoning",
        "steps": [
          {
            "arguments": [],
            "keyword": "Given ",
            "line": 6,
            "name": "If there is \"here\" and \"there\"",
            "match": {
              "location": "features/support/steps.js:9"
            },
            "result": {
              "status": "passed",
              "duration": 1546712
            }
          },
          {
            "arguments": [],
            "keyword": "Then ",
            "line": 7,
            "name": "I attain nirvana",
            "match": {
              "location": "features/support/steps.js:14"
            },
            "result": {
              "status": "passed",
              "duration": 154070
            }
          }
        ],
        "tags": [
          {
            "name": "@P0",
            "line": 1
          },
          {
            "name": "@NVTES-TC-33",
            "line": 4
          }
        ],
        "type": "scenario"
      }
    ],
    "id": "minimal-scenario",
    "line": 2,
    "keyword": "Feature",
    "name": "Minimal Scenario",
    "tags": [
      {
        "name": "@P0",
        "line": 1
      }
    ],
    "uri": "features/AIODemo.feature"
  }
]

Uploading results to AIO Tests

Post execution of a suite, the cucumber.json file can be uploaded either via

...

If the same file is uploaded again, the cases will be identified using the automation key (scenario.id)and would be updated, instead of creating new cases.

Reporting results via Cucumber Hooks and AIO Tests REST APIs

AIO Tests provides a rich set of APIs for Execution Management, using which users can not only report execution status, but also add effort, actual results, comments, defects and attachments to runs as well as steps.
AIO Tests also provides APIs to create cycles and to add cases to cycles for execution planning.

The basic sample below will show how Cucumber Hooks can leverage the AIO Tests REST APIs to report results.

...