Our documentation site has been updated to AIO Tests Knowledge Base

TestNG

AIO Tests allows easy reporting of TestNG results by supporting importing the testng-results.xml.

This article discusses how to import TestNG results and map it to tests existing in AIO Tests.

TestNG 

TestNG is a Java testing framework inspired by JUnit. TestNG strives to cover a wider range of test categories from unit to integration to end-to-end testing.

It is often, but not solely, combined with Selenium/Webdriver to drive Java based tests. With its powerful features of data driven test, parallel runs, flexible setup/teardown and inbuilt reporting, it has become one of the most used Java frameworks alongside JUnit.

Sample TestNG Report

TestNG offers an XML reporter capturing TestNG specific information like groups and data provider data.

Sample TestNG Report of a simple single test. The report gives information of total tests, groups of the tests, their statuses and duration.

<?xml version="1.0" encoding="UTF-8"?> <testng-results ignored="0" total="1" passed="1" failed="0" skipped="0"> <reporter-output> </reporter-output> <suite started-at="2020-08-04T15:08:54 IST" name="TngSuite" finished-at="2020-08-04T15:08:54 IST" duration-ms="46"> <groups> <group name="P0"> <method signature="ForestCreatorTests.testPlantTree()[pri:0, instance:com.aio.tests.ForestCreatorTests@74fe5c40]" name="testPlantTree" class="com.aio.tests.ForestCreatorTests"/> </group> <!-- P0 --> </groups> <test started-at="2020-08-04T15:08:54 IST" name="Testing first level" finished-at="2020-08-04T15:08:54 IST" duration-ms="46"> <class name="com.aio.tests.ForestCreatorTests"> <test-method signature="testPlantTree()[pri:0, instance:com.aio.tests.ForestCreatorTests@74fe5c40]" started-at="2020-08-04T15:08:54 IST" name="testPlantTree" finished-at="2020-08-04T15:08:54 IST" duration-ms="8" status="PASS"> <reporter-output> </reporter-output> </test-method> <!-- testPlantTree --> </class> <!-- com.aio.tests.ForestCreatorTests --> </test> </suite> </testng-results>

Generating TestNG XML Report

The XMLReporter is enabled by default in TestNG and no extra effort is required.

The report “testng-results.xml” is generated by default in the standard output directory, defaults to “test-output”.

If running through an IDE, check the settings for the TestNG configuration and enable default reporters if not enabled.

 

IntelliJ settings

 

Status Mapping TestNG → AIO Tests

TestNG

AIO Tests

TestNG

AIO Tests

PASS

Passed

FAIL

Failed

SKIP

Not Run

SKIP due to retry

(n-1) runs - Marked as failed [Note: TestNG shows them as skipped]
nth run - Status based on above 3 statuses.

Mapping automated TestNG tests to AIO Tests

In going with the simplicity advantage of AIO Tests, the TestNG integration has been designed as a simple and non-intrusive integration, without any extra dependency or coding required.

Groups of TestNG should be used to specify the key existing in AIO Tests.
However, there are many variations available in TestNG, which are captured in the xml report.

Below is a list of various scenarios that TestNG offers and a way to capture those in AIO Tests

1. Normal standalone case

Groups can be used to specify the AIO Tests key. In below example,
“AT-TC-25” has been added as a group, which maps to an existing case in AIO Tests.

public class ForestCreatorTests { @Test(groups = {"P0","AT-TC-25"}, description = "Verify positive case - all trees get planted") public void testPlantTree() { ForestCreator fc = new ForestCreator(); int plantedTrees = fc.plantTrees(10); Assert.assertEquals(plantedTrees, 10, "Verify all trees get planted"); } }

2. Mapping an automated case to multiple manual tests

Multiple manual case keys (in eg. below : "AT-TC-26","AT-TC-27","AT-TC-28") can be specified if an automated case covers more than one scenario. 3 cases in AIO Tests will be updated with the test result of this single test.

@Test(groups = {"P0","AT-TC-26","AT-TC-27","AT-TC-28"}, description = "Verify plant is healthy with correct process") public void testCreateForest() { ForestCreator fc = new ForestCreator(); fc.plantTrees(1); fc.waterThePlants(); fc.removeWeeds(); Assert.assertEquals(fc.getHealthyTreeCount(), 1, "Verify end to end plant growth"); }

3. Data Provider driven cases

There can be many variations to data provider cases:

3.a. Each data in dataprovider is one manual case. TestNG runs the below test 3 times with different data supplied by the Data Provider method. In this case, with each data, the corresponding manual case key needs to be specified eg. below shows dataprovider providing AT-TC-29 to 31.

3.b. All data point to same manual case. In this case, the AIO case key need not be sent with data, but can be added similar to normal cases in groups. In example below, AT-TC-33 has been added to @Test

AIO APIs expose a parameter, while uploading results, to specify whether multiple results of a case in a results.xml should be captured as separate runs of the same case or just capture the last run’s result. To know more, jump to Importing Results.

3.c. Mixed Bag Some data point to the same case, whereas some point to individual manual cases.

In this case, a combination of groups and data provider keys can be used. In example below, AT-TC-33 has been added to @Test and AT-TC-34 has been added with dataprovider data. In this case, Baobab and Banyan would report results for AT-TC-33 and “Palm” would report result to AT-TC-34

3.d. Dataprovider case mapping to multiple manual case

Comma separated case keys can be provided to map a single data provider driven case to multiple AIO tests. In example below, the result of “Baobab” would update two cases - AT-TC-40, AT-TC-42.

In total, these 3 tests would translate to 4 tests in Cycle executions.

4. Multiple invocation Counts

Same case gets executed multiple times. In below example, based on the option selected while uploading results, 5 runs can be created for AT-TC-25 or the last result can be captured in one run.

Mapping a case to a folder

Users have the option to designate a default folder for newly created cases. They can also define a mapping between a case and a folder by simply adding a group starting with the prefix @AIO-FOLDER- and mentioning the folder hierarchy. Folder hierarchy can be separated using / , for example - @AIO-FOLDER-Reports/Audit log
Note- This functionality is applicable only:

  • if a new case is generated through the import process

  • if the folder structure specified in the tag already exists.

    Additionally, if configured, this will supersede the default folder setting.

  1. Mapping using the default folder option

  1. Mapping each case using tags

 

Importing Results

Post execution of a TestNG suite, the testng-results.xml file can be uploaded either via

Please follow the above links to continue to import results using either of the options.