1 AIO Tests Playwright Reporter
2 How does the AIO Tests Reporter work
3 How to get started?
4 Use
- 4.1 Mapping automated Playwright tests to AIO Tests
- 4.2 Setup
  - 4.2.1 Cloud - Authentication
  - 4.2.2 Jira Server/Data Center - Authentication
  - 4.2.3 Configurable values
  - 4.2.4 Create New Cycle options
  - 4.2.5 Setting browser or project name in run custom fields
  - 4.2.6 Parallel builds
5 Logging

AIO Tests Playwright Reporter

AIO Tests for Jira is a Jira-native test management app covering the entire QA lifecycle with Test Cases, Cycles, Reports, Automation, Dashboards, providing a one stop shop for all your testing needs.

With it's Playwright reporter, AIO Tests simplifies reporting of results from the automated Playwright tests to AIO Tests for Jira.

How does the AIO Tests Reporter work

By hooking into Playwright events, the AIO Tests Reporter reports results in the onExit event, ater all the tests are run.
The reporter can create a new cycle for the executions or reuse existing cycles, based on the configuration done in playwright.config.ts.
It can also upload attachments for failed executions.
Retries can either be reported as new runs or used to update the existing execution run.

How to get started?

npm install aiotests-playwright-reporter

Use

Mapping automated Playwright tests to AIO Tests

The AIO Tests Case key can be added to tags of Playwright cases or they can be added to the test title with @ key to mark it as a Playwright tag.
If there are multiple case keys added as tags, then the result of one test will be updated to multiple cases.

✒ Please note that the case key can appear anywhere in the description

Mapping single case

   test('should clear text input field when an item is added',{tag:'@NVTES-TC-1'}, async ({ page }) => {
   //Or test('@NVTES-TC-1 : should clear text input field when an item is added', async ({ page }) => {
    const newTodo = page.getByPlaceholder('What needs to be done?');
    await newTodo.fill(TODO_ITEMS[0]);
    await newTodo.press('Enter');
    await expect(newTodo).toBeEmpty();
    await checkNumberOfTodosInLocalStorage(page, 1);
  });

Mapping to multiple cases

   test('should clear text input field when an item is added',{tag: ['@NVTES-TC-4','@NVTES-TC-5']}, async ({ page }) => {
    const newTodo = page.getByPlaceholder('What needs to be done?');
    await newTodo.fill(TODO_ITEMS[0]);
    await newTodo.press('Enter');
    await expect(newTodo).toBeEmpty();
    await checkNumberOfTodosInLocalStorage(page, 1);
  });

Setup

const aioConfigDetails = {configDetailedBelow};

module.exports = defineConfig({
....
reporter: [['aiotests-playwright-reporter',{ aioConfig: aioConfigDetails}]],

Cloud - Authentication

For Jira Cloud (eg. https://yourco.atlassian.net/..), the "cloud" property needs to be set in the config for authentication.

Local runs : The API Key generated from AIO Tests, needs to be set as "apiKey" value.
CI/CD: For CI runs, you can set the AIO_API_KEY environment variable to pass it externally as a SECRET.

  
    const aioConfigDetails = {
      "enableReporting": true,
      "cloud": {
        "apiKey": "<your API KEY>"
      },
      "jiraProjectId": "SCRUM",
      "cycleDetails": {
        "createNewCycle": true, //possible values "true","false","CREATE_IF_ABSENT", true, false
        "cycleName": "Playwright Nightly Run ", //used if createNewCycle set to true or "CREATE_IF_ABSENT"
        "cycleKey": "SERV-CY-2", //used if createNewCycle set to false or "false"
        "folder": ["Cloud","Smoke Test Nightly"], //array of folder hierarchy
        "tasks": ["SCRUM-1","SCRUM-2],
        "customFields": [ //optional
          {
            "name": "Reviewed? [Boolean CF]",
            "value": "Yes",
          },
          {
            "name": "Set (Single Select CF)",
            "value": {"value":"P1"}
          },
          {
            "name": "Teams (Multi Select CF)",
            "value": [{"value":"TeamAlpha"},{"value":"Zeta"}]
          },
          {
            "name": "NumberCF", "value": 0
          },
          {
            "name": "TextValue CF", "value": "This can be a long note",
          },
          {
            "name": "Reviewed Date", "value": "2024-08-29T04:38:36.437Z",
          },
          {
            "name": "SME [User CF]", "value": "<accountid of user>"
          }
        ],
      },
       "runDetails": { //optional
        "customFieldsToUpdate": [
          {
            "operationType": "ADD_TO_EXISTING",
            "name": "StepOwner",
            "value": [{"value": "Val2"},{"value": "Val1"}]
          },
          {
            "name": "Env", "value": {"value":"UAT"}
          },
          {
            "name": "Browser", "value": {"value":"$project"}
          }
        ]
      },
      "addNewRun": true,
      "addAttachmentToFailedCases": true,
      "createNewRunForRetries": true,
      "debugMode": false,
      "parallelBuild":{ //optional
        "masterBuild": true,
        "waitForSeconds": 10 //defaults to 2 seconds
      }
    }
  }
}

Jira Server/Data Center - Authentication

For Jira Hosted or DataCenter versions, the "hosted" property needs to be set in the for authentication as shown below.
The "jiraUrl" needs to be specified with the base url of the hosted Jira instance.

Authentication is supported either by providing Jira username and password or by using the Jira PAT. More information can be found on Server Authentication here

Local runs : For local runs, either "jiraUsername" + "jiraPassword" can be set or one can simply set the "jiraPAT" value.
CI/CD: For CI runs, you can set the JIRA_USERNAME and JIRA_PASSWORD or JIRA_PATenvironment variable to pass it externally as a SECRET.

{
    "enableReporting": true,
    "hosted" : {
      "jiraUrl": "https://jira.yourco.com",
      "jiraPAT": "PAT from Jira Tokens | JIRA_PAT as environment variable",
      "jiraUsername": "Jira Username. If PAT is specified, then username is not required",
      "jiraPassword": "Jira password, required if authentication is through username/password"
    },
    "jiraProjectId": "SERV",
    "cycleDetails": {
      "createNewCycle": true, //possible values "true","false","CREATE_IF_ABSENT", true, false
      "cycleName": "Playwright Nightly Run ", //used if createNewCycle set to true or "CREATE_IF_ABSENT"
      "cycleKey": "SERV-CY-2", //used if createNewCycle set to false or "false"
      "folder": ["Server","Smoke Test Nightly"],//array of folder hierarchy
      "tasks": ["SERV-1","SERV-2],
      "customFields": [ //optional
        {
          "name": "Reviewed? [Boolean CF]",
          "value": "Yes",
        },
        {
          "name": "Set (Single Select CF)",
          "value": {"value":"P1"}
        },
        {
          "name": "Teams (Multi Select CF)",
          "value": [{"value":"TeamAlpha"},{"value":"Zeta"}]
        },
        {
          "name": "NumberCF", "value": 0
        },
        {
          "name": "TextValue CF", "value": "This can be a long note",
        },
        {
          "name": "Reviewed Date", "value": "2024-08-29T04:38:36.437Z",
        },
        {
          "name": "SME [User CF]", "value": "<accountid of user>"
        }
      ],
    },
    "runDetails": { //optional
      "customFieldsToUpdate": [
        {
          "operationType": "ADD_TO_EXISTING",
          "name": "StepOwner",
          "value": [{"value": "Val2"},{"value": "Val1"}]
        },
        {
          "name": "Env", "value": {"value":"UAT"}
        },
        {
          "name": "Browser", "value": {"value":"$project"}
        }
      ]
    },
    "addNewRun": true,
    "addAttachmentToFailedCases": false,
    "createNewRunForRetries": false,
    "addTestBodyToComments": true,
    "debugMode": false,
    "parallelBuild":{ //optional
      "masterBuild": true,
      "waitForSeconds": 10 //defaults to 2 seconds
    }
}

Configurable values

Value	Description

Value	Description
enableReporting	Set to true to make the current run update results to AIO Tests. Default false.
jiraProjectId	Jira Project key to update results to
cycleDetails.createNewCycle	Options: [true, false, "CREATE_IF_ABSENT"]. Set to true to create a new cycle for run being reported.
cycleDetails.cycleName	Works if createNewCycle is true, sets the cycle name of cycle getting created
cycleDetails.cycleKey	AIO Tests cycle key that should be updated. Used if createNewCycle is false
cycleDetails.folder	Folder hierarchy, where first item in array is parent folder and so on eg.["Parent","Child"]
cycleDetails.tasks	List of Jira Issue Keys to attach as Tasks to created cycle, impacts only when creating new cycle
cycleDetails.customFields	List of custom fields that need to be set while creating cycle. Options shown in example.
addNewRun	Create a new run or update an existing run in the cycle
addAttachmentToFailedCases	Set to true to attach screenshots, if available, for failed cases
createNewRunForRetries	Set to true if each retry should create a new run
newRunForRepeatedTest	Set to true test script body should be added as a comment in a failed case. Doesn't work above v12.x
runDetails.customFieldsToUpdate	Set to true each repeated test would create a new run
customFieldsToUpdate.operationType	Options: ADD_TO_EXISTING, REPLACE_EXISTING, DELETE_EXISTING
debugMode	Default false. Set to true to increase verbosity of logs while debugging an issue
parallelBuild.masterBuild	Optional. Default true. See below for details on parallelBuild
parallelBuild.waitForSeconds	Optional. Default 2 seconds. See below for details on parallelBuild

Create New Cycle options

createNewCycle = true or "true", uses the cycleName and cycleDetails value to generate new cycle
createNewCycle = false or "false", uses the cycleKey value to find an existing cycle and updates the cycle. If cycle is not found, an error is thrown
createNewCycle = "CREATE_IF_ABSENT" uses the cycleName to search for an existing cycle with an exact match. If cycle is found, then the cycle is updated. If it is not found, then a new cycle is created using cycleName and cycleDetails

Setting browser or project name in run custom fields

Since Playwright supports running cases on multiple browsers, there may be a need to capture the browser details where the run has executed in AIO Tests. The browser value is set as the project name in playwright config file. The project name can be set in the custom fields in AIO as below:

Setup a run custom field in AIO (Project Settings -> Custom Fields)
In the aioConfig, specify it in the runDetails.customFieldsToUpdate property as below:
(the example below assumes the run custom field is named "Browser" and the values if defined as a single select list, match the text used in the project name in playwright config)

...
"runDetails": { 
      "customFieldsToUpdate": [
        {
          "name": "Browser", "value": {"value":"$project"}
        }
      ]
    }
    ...

Parallel builds

If multiple builds are being triggered in parallel, the parallelBuild setting can be used to specify the masterBuild.
One of the parallel builds can be configured to set masterBuild as true. This build should have createNewCycle either set as true or CREATE_IF_ABSENT.
The other builds running in parallel can have masterBuild set to false, which would imply, they would wait for the masterBuild to run the cycle creation code, waiting for waitForSeconds (defaults to 2 seconds), before trying to find the cycle.

If multiple builds are not being run in parallel, the parallelBuild value can be ignored.

Logging

AIO Tests Reporter logs can be seen in the run logs as below for successful updates

*************** AIO Tests Reporter AIO Tests Playwright Reporter - Reporting results - test 2 ***************
11:38:10 AM : Debug Mode is set to true
*************** AIO Tests Reporter : Determining cycle to update ***************
11:38:17 AM : Reporting results to cycle : NVTES-CY-828
*************** AIO Tests Reporter : Reporting results ***************
11:38:17 AM : ***** Updating failed cases *****
11:38:17 AM : Posting results to /project/NVTES/testcycle/NVTES-CY-828/testcase/NVTES-TC-4/testrun?createNewRun=true
11:38:20 AM : Successfully reported NVTES-TC-4 as Failed with runID 11362947.
11:38:22 AM : ***** Updating all passed cases *****
11:38:22 AM : Reporting results in bulk: /project/NVTES/testcycle/NVTES-CY-828/bulk/testrun/update?createNewRun=true
11:38:23 AM : Successfully reported 2 passed cases for browser: chromium.
*************** AIO Tests Reporter : AIO Tests Playwright Reporter - Reporting results completed ***************

Errors received while updating will appear in a similar way

*************** AIO Tests Reporter  Reporting results ***************
***** Updating all passed cases *****
Successfully reported 0 passed cases for browser: chromium.
 - Failures in reporting passed cases:
 - 120801: aio.jira.tcms.exceptions.InvalidDataException: The selected version for Case NVTES-TC-20 is in Draft status. This status is not eligible for addition to a cycle.
 - 120802: aio.jira.tcms.exceptions.InvalidDataException: The selected version for Case NVTES-TC-21 is in Draft status. This status is not eligible for addition to a cycle.
Successfully reported 0 passed cases for browser: firefox.
 - Failures in reporting passed cases:
 - 120801: aio.jira.tcms.exceptions.InvalidDataException: The selected version for Case NVTES-TC-20 is in Draft status. This status is not eligible for addition to a cycle.
 - 120802: aio.jira.tcms.exceptions.InvalidDataException: The selected version for Case NVTES-TC-21 is in Draft status. This status is not eligible for addition to a cycle.
*************** AIO Tests Reporter : AIO Tests Playwright Reporter - Reporting results completed ***************

To troubleshoot an issue, debugMode can be set to true in config, to increase logging verbosity and to see detailed response errors

AIO Tests Knowledge Base

Playwright AIO Tests Reporter