Content Comparison

...

JUnit is one of the most widely used open-source Unit Testing Framework frameworks for JAVA and has been a key player in promoting test-driven development.

The latest version of JUnit, JUnit 5 has introduced many new features, new APIs, and the concept of a test engine to simplify test automation (e.g. @DisplayName, @BeforeXX, @ParametrizedTest, etc.).

...

Being one of the earliest testing frameworks and with a huge community base, JUnit’s XML report has got gained widespread adoption. Many continuous integration servers accept JUnit XML reports as their de facto standard for reporting test results.

The report format gradually became ubiquitous for test results reporting and frameworks across languages like PyTest, Cucumber, Fitnesse, RSpec, Katalon, etc. All these have reporting extensions which that generate the JUnit XML report, which implies cases from any of these frameworks can be imported into AIO Tests.

Sample JUnit XML Report

...

Expand
title TEST-FeatureTests.xml
Code Block

Generating JUnit XML Report

Maven Surefire

Maven surefire plugin is required to generate the JUnit XML Report.

JUnit 4 Settings

...

title	JUnit 4 pom.xml setting

...

JUnit 5 Settings

If the JUnit 5 features are being utilized, the junit-jupiter-engine needs to be added with additional configurations for fine-grained configuration of reports.
Tests need to be annotated with @DisplayName and <usePhrasedXXXName>true</usePhrasedXXXName> fields need to be set to true as shown below.

For more info, refer Maven Surefire - Using JUnit 5

...

title	JUnit 5 - Support for DisplayNames - Value in DisplayName would come as Method name.

...

Ant

If you are using Ant, the following tasks need to be used to generate the report.

JUnit 4 - JUnit Task

...

title	Task JUnit + JUnit report

...

JUnit 5 - JUnitlauncher Task

Using the junitlauncher task allows launching the JUnit 5 test launcher and building the test requests to be executed by the test engine(s) supported by JUnit 5. For more information, please refer JUnitLauncher.

...

title	JUnit 5 ANT task

...

JUnit XML

...

Description

...

AIO Tests Mapping

...

No tag inside <testcase> means Passed

...

Passed case

...

Passed

...

</skipped>

...

Skipped case either by @Ignore or others

...

Not Run

...

</failure>

...

Indicates that the test failed. A failure is a test which the code has explicitly failed by using the mechanisms for that purpose. e.g., via an assertEquals.

...

Failed

...

</error>

...

Indicates that the test errored. An errored test is one that had an unanticipated problem. e.g., an unchecked throwable;

...

Failed

...

In going with the simplicity advantage of AIO Tests, the JUnit integration has been designed as a simple and non-intrusive integration, without any extra dependency or coding required.

...

There are 4 ways that a JUnit case is mapped to an AIO Case.

JUnit 5: @DisplayName gives a way to specify generic information in the test name instead of the hard rule based Java method names. This can be used to specify an existing AIO Case Key.
For example, @DisplayName("SCRUM-TC-1973 : Test forest gets created.")
Testcase key in Underscores : Since Java and Python, do not allow hyphens in their method signatures, AIO Tests allows case keys with underscores as valid mapping. The AIO Test key can be used in the method name by replacing the hyphen with underscores. eg. if AT-TC-123 is being automated by a test, the test method can be named as verifyNotNullType_AT_TC_123 and the results of this method will be marked against AT-TC-123.
Automation Key: The mapping of an AIO Case to an automated JUnit case can happen through a field Automation Key, which can be specified in the Case in AIO Tests app as shown below. The automation key is the fully qualified name of the test method i.e packagename.classname.methodname.
This can be useful at places where you do not want to add AIO case keys in the name or annotations or where automation is happening first without a manual case being created in AIO Tests.
Image Removed
New Case creation: If no case key (e.g. SCRUM-TC-xx) or automation key is found with existing cases, a new case is created with
- title as the name value from <testcase> tag of the JUnit report
- automation key as classname.name from the JUnit report.
- status as Published
- automation status as Automated
- automation owner as user uploading the results
The following flow summarizes the logic of JUnit case mapping when a key is not found via @DisplayName and via Automation key
Image Removed

Examples

Below are few examples that show results on testing the ForestCreator.java class.

Mapping using JUnit5 @DisplayName (when settings are done as shown above for DisplayName in SureFire/ANT task)

🖊 Add case key to Display name

Code Block

@DisplayName("SCRUM-TC-2309 : Verify positive case - all trees get planted")
@Test
public void testPlantTree() {
    ForestCreator fc = new ForestCreator();
    int plantedTrees = fc.plantTrees(10);
    Assertions.assertEquals(plantedTrees, 10, "Verify all trees get planted");
}

Report:
<testcase name="SCRUM-TC-2309 : Verify positive case - all trees get planted" 
classname="com.aio.tests.junit5.ForestCreatorTests" time="0.001">

Output: On import of the above report in AIO Tests, SCRUM-TC-2309 is updated. If case doesn’t exist then an appropriate error would be thrown

Mapping one automated case with multiple manual cases with JUnit5 @DisplayName

🖊 Add all case keys as part of Display name

Code Block

@DisplayName("SCRUM-TC-2307, SCRUM-TC-2308,SCRUM-TC-2350 : Verify positive case - all trees get planted")
@Test
public void testCreateForest() {
    ForestCreator fc = new ForestCreator();
    fc.plantTrees(1);
    fc.waterThePlants();
    fc.removeWeeds();
    Assertions.assertEquals(fc.getHealthyTreeCount(), 1, "Verify end to end plant growth");
}

Report:
<testcase name="SCRUM-TC-2307, SCRUM-TC-2308,SCRUM-TC-2350 : 
Verify positive case - all trees get planted" 
classname="com.aio.tests.junit5.ForestCreatorTests" time="0.001">

Output: On import of the above report in AIO Tests, 3 cases are updated with result of the above execution → SCRUM-TC-2307, SCRUM-TC-2308 and SCRUM-TC-2350. If case doesn’t exist then an appropriate error would be thrown

Mapping JUnit 4 case to existing manual case(s)

🖊 Method name can contain case key with underscores : plantTree_should_returnNumberOfTrees_AT_TC_222. . OR

🖊 Automation Owner would have to mark the automated key value in an existing case to com.aio.tests.junit4.ForestCreatorTests.plantTree_should_returnNumberOfTrees

🖊 If single automated JUnit case needs to update multiple cases, then the automated key needs to be updated in multiple AIO Tests automation key values

Code Block

@Test
public void plantTree_should_returnNumberOfTrees() {
    ForestCreator fc = new ForestCreator();
    int plantedTrees = fc.plantTrees(10);
    Assertions.assertEquals(plantedTrees, 10, "Verify all trees get planted");
}

Report:
<testcase name="plantTree_should_returnNumberOfTrees" 
classname="com.aio.tests.junit4.ForestCreatorTests" time="0.001">

Output: On import of the above report in AIO Tests, the automation key with com.aio.tests.junit4.ForestCreatorTests.plantTree_should_returnNumberOfTrees would be searched for and the cases marked with this key would be added to cycle and updated

Parametrized Named Cases Convention

With the release of the data driven feature, it is now possible to create cases with datasets from JUnit cases.

The convention for creating data parameters with names is (params: parameterName1={0}, parameterName2={1}, parameterName1={2})

eg.

(params: userType={0},profile={1},income={2}). This example would create a case with 3 data parameters : userType, profile and income and create datasets based on the Junit source values

The below example would create a testcase with 5 datasets and a data parameter named month

...

title	JUnit 5

...

title	JUnit 4

...

🖊 A new flag has been introduced updateDatassets. This flag controls, whether the change in datasets should be used to update the Junit datasets case or not. The value can be set to false, when a subset of datasets are being run to avoid removal of datasets. If set to true, this flag affects only existing datasets cases.

Parametrized Non-Named Cases

🖊 If the name is not specified, and a default report is generated, AIO Tests generates new parameter names for JUnit 5 reports, which elaborate on user data. However, in Junit 4, in the absence of data in the report, cases would continue to work as individual cases.

In below case, AIO Tests would create a parameter named Parameter 1 and would create 5 datasets with the numbers [1, 3, 5, -3 and 15] as values.

Expand

title	JUnit 5 Default report sample

Code Block

<?xml version="1.0" encoding="UTF-8"?>
<testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report-3.0.xsd" version="3.0" name="com.aio.tests.junit5.ForestCreatorTests" time="0.003" tests="2" errors="0" skipped="0" failures="1">
  <properties>
    <property name="gopherProxySet" value="false"/>
    ....
  </properties>
  <testcase name="AT-TC-26, AT-TC-27,AT-TC-28 : Verify positive case - all trees get planted" classname="com.aio.tests.junit5.ForestCreatorTests" time="0.001">
    <failure type="org.opentest4j.AssertionFailedError"><![CDATA[org.opentest4j.AssertionFailedError: Verify end to end plant growth ==> expected: <10> but was: <1>
   at com.aio.tests.junit5.ForestCreatorTests.testCreateForest(ForestCreatorTests.java:24)
]]></failure>
    <system-out><![CDATA[Planting tree 0
Watering the plants 
Removing the weeds 
]]></system-out>
  </testcase>
  <testcase name="SCRUM-TC-25 : Verify positive case - all trees 
  get planted" classname="com.aio.tests.junit5.ForestCreatorTests" time="0.001">
    <system-out><![CDATA[Planting tree 0
...
]]></system-out>
  </testcase>
</testsuite>

Generating JUnit XML Report

Maven Surefire

The Maven surefire plugin is required to generate the JUnit XML Report.

JUnit 4 Settings

Expand

title	JUnit 4 pom.xml setting

Code Block
<build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>3.0.0-M5</version> </plugin> ... </plugins> </build>

JUnit 5 Settings

If the JUnit 5 features are being utilized, the junit-jupiter-engine needs to be added with additional configurations for fine-grained configuration of reports.
Tests need to be annotated with @DisplayName and <usePhrasedXXXName>true</usePhrasedXXXName> fields need to be set to true as shown below.

For more info, refer to Maven Surefire - Using JUnit 5

Expand

title	JUnit 5 - Support for DisplayNames - Value in DisplayName would come as Method name.

Code Block

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>3.0.0-M5</version>
    <dependencies>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <version>5.7.0-M1</version>
        </dependency>
    </dependencies>
    <configuration>
                <statelessTestsetReporter implementation="org.apache.maven.plugin.surefire.extensions.junit5.JUnit5Xml30StatelessReporter">
                    <disable>false</disable>
                    <version>3.0</version>
                    <usePhrasedFileName>true</usePhrasedFileName>
                    <usePhrasedTestSuiteClassName>true</usePhrasedTestSuiteClassName>
                    <usePhrasedTestCaseClassName>true</usePhrasedTestCaseClassName>
                    <usePhrasedTestCaseMethodName>true</usePhrasedTestCaseMethodName>
                </statelessTestsetReporter>
.....
      </configuration>
 </plugin>

Ant

If you are using Ant, the following tasks need to be used to generate the report.

JUnit 4 - JUnit Task

Expand

title	Task JUnit + JUnit report

Code Block

<target name = "test" depends = "compile">
  <mkdir dir="reports/raw"/>
  <junit>
      <test name="my.test.TestCase" outfile="result" todir="reports/raw">
      <formatter type="xml"/>
      </test>
  </junit>
</target>
<target name="test-reports" depends="test">
  <junitreport todir="./reports">
      <fileset dir="reports/raw/">
          <include name="TEST-*.xml"/>
      </fileset>
      <report format="frames" todir="./report/html"/>
  </junitreport>
</target>

JUnit 5 - JUnitlauncher Task

Using the junitlauncher task allows launching the JUnit 5 test launcher and building the test requests to be executed by the test engine(s) supported by JUnit 5. For more information, please refer to JUnitLauncher.

Expand

title	JUnit 5 ANT task

Code Block

<junitlauncher>
    <!-- include the JUnit platform related libraries
    required to run the tests -->
    <classpath refid="junit.platform.libs.classpath"/>

    <!-- include the JUnit Jupiter engine libraries -->
    <classpath refid="junit.engine.jupiter.classpath"/>

    <classpath>
        <!-- the test classes themselves -->
        <pathelement location="${build.classes.dir}"/>
    </classpath>
    <testclasses outputdir="${output.dir}">
        <fileset dir="${build.classes.dir}"/>
        <listener type="legacy-xml" sendSysErr="true" sendSysOut="true"/>

    </testclasses>
</junitlauncher>

XML generated by this formatter can be used as-is by the junitreport task.

Status Mapping JUnit → AIO Tests

JUnit XML	Description	AIO Tests Mapping
No tag inside <testcase> means Passed	Passed case	Passed
</skipped>	Skipped case either by @Ignore or others	Not Run
</failure>	`Indicates that the test failed. A failure is a test which the code has explicitly failed by using the mechanisms for that purpose. e.g., via an assertEquals.`	Failed
</error>	`Indicates that the test errored. An errored test is one that had an unanticipated problem. e.g., an unchecked throwable;`	Failed

Mapping automated JUnit tests to AIO Tests

In going with the simplicity advantage of AIO Tests, the JUnit integration has been designed as a simple and non-intrusive integration, without any extra dependency or coding required.

...

There are 4 ways that a JUnit case is mapped to an AIO Case.

JUnit 5: @DisplayName gives a way to specify generic information in the test name instead of the hard rule-based Java method names. This can be used to specify an existing AIO Case Key.
For example, @DisplayName("SCRUM-TC-1973 : Test forest gets created.")
Testcase Key in Underscores: Since Java and Python, do not allow hyphens in their method signatures, AIO Tests allows case keys with underscores as valid mapping. The AIO Test key can be used in the method name by replacing the hyphen with underscores. eg. if AT-TC-123 is being automated by a test, the test method can be named verifyNotNullType_AT_TC_123 and the results of this method will be marked against AT-TC-123.
Automation Key: The mapping of an AIO Case to an automated JUnit case can happen through a field Automation Key, which can be specified in the Case in AIO Tests app as shown below. The automation key is the fully qualified name of the test method i.e. packagename.classname.methodname.

Info
This can be useful at places where you do not want to add AIO case keys in the name or annotations or where automation is happening first without a manual case being created in AIO Tests.

...

New Case Creation: If no case key (e.g. SCRUM-TC-xx) or automation key is found with existing cases, a new case is created with:
- Title as the name value from <testcase> tag of the JUnit report
- Automation key as classname.name from the JUnit report.
- Status as Published
- Automation status as Automated
- Automation owner as a user uploading the results

The following flow summarizes the logic of JUnit case mapping when a key is not found via @DisplayName and via the Automation key.

...

Examples

Below are a few examples that show the results of testing the ForestCreator.java class.

Mapping using JUnit5 @DisplayName (when settings are done as shown above for DisplayName in SureFire/ANT task)

🖊 Add case key to Display name

Code Block

@DisplayName("SCRUM-TC-2309 : Verify positive case - all trees get planted")
@Test
public void testPlantTree() {
    ForestCreator fc = new ForestCreator();
    int plantedTrees = fc.plantTrees(10);
    Assertions.assertEquals(plantedTrees, 10, "Verify all trees get planted");
}

Report:
<testcase name="SCRUM-TC-2309 : Verify positive case - all trees get planted" 
classname="com.aio.tests.junit5.ForestCreatorTests" time="0.001">

Output: On import of the above report in AIO Tests, SCRUM-TC-2309 is updated. If case doesn’t exist then an appropriate error would be thrown

Mapping one automated case with multiple manual cases with JUnit5 @DisplayName

🖊 Add all case keys as part of the Display name

Code Block

@DisplayName("SCRUM-TC-2307, SCRUM-TC-2308,SCRUM-TC-2350 : Verify positive case - all trees get planted")
@Test
public void testCreateForest() {
    ForestCreator fc = new ForestCreator();
    fc.plantTrees(1);
    fc.waterThePlants();
    fc.removeWeeds();
    Assertions.assertEquals(fc.getHealthyTreeCount(), 1, "Verify end to end plant growth");
}

Report:
<testcase name="SCRUM-TC-2307, SCRUM-TC-2308,SCRUM-TC-2350 : 
Verify positive case - all trees get planted" 
classname="com.aio.tests.junit5.ForestCreatorTests" time="0.001">

Output: On import of the above report in AIO Tests, 3 cases are updated with the result of the above execution → SCRUM-TC-2307, SCRUM-TC-2308 and SCRUM-TC-2350. If the case doesn’t exist then an appropriate error would be thrown

Mapping JUnit 4 case to existing manual case(s)

🖊 Method name can contain case key with underscores:

plantTree_should_returnNumberOfTrees_AT_TC_222.OR

🖊 The Automation Owner would have to mark the automated key value in an existing case to com.aio.tests.junit4.ForestCreatorTests.plantTree_should_returnNumberOfTrees

🖊 If a single automated JUnit case needs to update multiple cases, then the automated key needs to be updated in multiple AIO Tests automation key values.

Code Block

@Test
public void plantTree_should_returnNumberOfTrees() {
    ForestCreator fc = new ForestCreator();
    int plantedTrees = fc.plantTrees(10);
    Assertions.assertEquals(plantedTrees, 10, "Verify all trees get planted");
}

Report:
<testcase name="plantTree_should_returnNumberOfTrees" 
classname="com.aio.tests.junit4.ForestCreatorTests" time="0.001">

Output: On import of the above report in AIO Tests, the automation key with com.aio.tests.junit4.ForestCreatorTests.plantTree_should_returnNumberOfTrees would be searched for and the cases marked with this key would be added to the cycle and updated.

Parametrized Named Cases Convention

With the release of the data-driven feature, it is now possible to create cases with datasets from JUnit cases.

The convention for creating data parameters with names is (params: parameterName1={0}, parameterName2={1}, parameterName1={2})

eg.

(params: userType={0},profile={1},income={2}). This example would create a case with 3 data parameters: userType, profile, and income and create datasets based on the Junit source values

The below example would create a test case with 5 datasets and a data parameter named month.

Expand

title	JUnit 5

Code Block

@ParameterizedTest(name="(params: month={0})")
@EnumSource(value=Month.class, names = {"APRIL", "JUNE", "SEPTEMBER", "NOVEMBER", "FEBRUARY"},
        mode = EnumSource.Mode.INCLUDE) 
void getValueForAMonth_IsAlwaysBetweenOneAndTwelve(Month month) {
    int monthNumber = month.getValue();
    assertTrue(monthNumber >= 1 && monthNumber <= 9);
}

Expand

title	JUnit 4

Code Block

@Parameterized.Parameters(name="(params : numberA={0},numberB={1})")
public static Collection<Object[]> data() {
    Object[][] obj = new Object[2][3];
    Random r = new Random();
    for (int i = 0; i < 2; i++) {
        Integer[] insert = {r.nextInt(10), r.nextInt(10), r.nextInt(20)};
        obj[i] = insert;
    }
    return Arrays.asList(obj);
}

@Test
public void test_addTwoNumbers() {
    assertEquals(calculator.add(numberA, numberB), expected);
}

🖊 A new flag has been introduced updateDatassets. This flag controls, whether the change in datasets should be used to update the Junit datasets case or not. The value can be set to false, when a subset of datasets are being run to avoid removal of datasets. If set to true, this flag affects only existing datasets cases.

Parametrized Non-Named Cases

🖊 If the name is not specified, and a default report is generated, AIO Tests generates new parameter names for JUnit 5 reports, which elaborate on user data. However, on JUnit 4, in the absence of data in the report, cases would continue to work as individual cases.

In the below case, AIO Tests would create a parameter named Parameter 1 and would create 5 datasets with the numbers [1, 3, 5, -3, and 15] as values.

Expand

title	JUnit 5 Default report sample

Code Block

<testcase name="isOdd_ShouldReturnTrueForOddNumbers(int)[1] 1" classname="com.aiodemos.junit5.DataDrivenTests" time="0.0"/>
  <testcase name="isOdd_ShouldReturnTrueForOddNumbers(int)[2] 3" classname="com.aiodemos.junit5.DataDrivenTests" time="0.001"/>
  <testcase name="isOdd_ShouldReturnTrueForOddNumbers(int)[3] 5" classname="com.aiodemos.junit5.DataDrivenTests" time="0.001"/>
  <testcase name="isOdd_ShouldReturnTrueForOddNumbers(int)[4] -3" classname="com.aiodemos.junit5.DataDrivenTests" time="0.001"/>
  <testcase name="isOdd_ShouldReturnTrueForOddNumbers(int)[5] 15" classname="com.aiodemos.junit5.DataDrivenTests" time="0.0"/>

Code Block

@RunWith(Parameterized.class)
public class ParametrizedRunTest {

  @Parameterized.Parameters
  public static Iterable<Object[]> planTrees() {
      return Arrays.asList(new Object[][] {
              { "Baobab" }, 
              { "Rainbow Eucalyptus" },
              { "Banyan" }
      });
  }
  
Report:  
<testcase name="plantTree[0]" classname="com.aio.tests.ParametrizedRunNonNamedTest" time="0"/>
<testcase name="plantTree[1]" classname="com.aio.tests.ParametrizedRunNonNamedTest" time="0">
    <failure message="asdf" type="org.junit.ComparisonFailure"><![CDATA[org.junit.ComparisonFailure: expected:<[BaobabTree]> but was:<[Rainbow Eucalyptus]>
   at com.aio.tests.ParametrizedRunNonNamedTest.plantTree(ParametrizedRunNonNamedTest.java:33)
]]></failure>
</testcase>
<testcase name="plantTree[2]" classname="com.aio.tests.ParametrizedRunNonNamedTest" time="0"/>

...

Post execution of a JUnit suite, the TEST-<xxx>.xml file can be uploaded either via

AIO Tests REST API call calls using multipart form - data to upload file
Cycle options in the Cycles Tab inside AIO Tests

Please follow the above links to continue to import results using either of the options.

...

Version	Old Version 1	New Version Current
Changes made by	APPWRK Writer (Unlicensed)	APPWRK Writer (Unlicensed)
Saved on	Feb 22, 2024	Feb 27, 2024

Content Comparison

Versions Compared

Key

Sample JUnit XML Report

ExpandtitleTEST-FeatureTests.xml Code Block

Generating JUnit XML Report

Maven Surefire

JUnit 4 Settings

JUnit 5 Settings

Ant

JUnit 4 - JUnit Task

JUnit 5 - JUnitlauncher Task

Examples

Mapping using JUnit5 @DisplayName (when settings are done as shown above for DisplayName in SureFire/ANT task)

Mapping one automated case with multiple manual cases with JUnit5 @DisplayName

Mapping JUnit 4 case to existing manual case(s)

Parametrized Named Cases Convention

Parametrized Non-Named Cases

Generating JUnit XML Report

Maven Surefire

JUnit 4 Settings

JUnit 5 Settings

Ant

JUnit 4 - JUnit Task

JUnit 5 - JUnitlauncher Task

Status Mapping JUnit → AIO Tests

Mapping automated JUnit tests to AIO Tests

Examples

Mapping using JUnit5 @DisplayName (when settings are done as shown above for DisplayName in SureFire/ANT task)

Mapping one automated case with multiple manual cases with JUnit5 @DisplayName

Mapping JUnit 4 case to existing manual case(s)

Parametrized Named Cases Convention

Parametrized Non-Named Cases

Expand
title TEST-FeatureTests.xml
Code Block