AIO Tests allows easy reporting of JUnit results by supporting the import of the TEST-feature-results.xml (the output file of a JUnit execution).
With AIO Tests JUnit integration, developers will not need to write manual cases inside AIO Tests. They can write JUnit cases which will be imported (created) in AIO Tests along with their execution results. This results in a holistic view of coverage of requirements (unit to manual to functional and end-to-end automated tests).
This article discusses how to import JUnit XML results and map it to existing cases in AIO Tests or generate new cases in AIO, in the absence of existing cases.
JUnit
JUnit is one of the most widely used open source Unit Testing Framework for JAVA and has been a key player in promoting test driven development.
The latest version of JUnit, JUnit 5 has introduced many new features, new APIs and the concept of test engine to simplify test automation (e.g. @DisplayName, @BeforeXX, @ParametrizedTest, etc.).
JUnit Reporting
JUnit’s XML report comes through the Maven Surefire plugin or through an ANT task.
Being one of the earliest testing frameworks and with a huge community base, JUnit’s XML report has got widespread adoption. Many continuous integration servers accept JUnit XML report as their de facto standard for reporting test results.
The report format gradually became ubiquitous for test results reporting and frameworks across languages like PyTest, Cucumber, Fitnesse, RSpec, Katalon etc. All these have reporting extensions which generate the JUnit XML report, which implies cases from any of these frameworks can be imported into AIO Tests.
Sample JUnit XML Report
Generating JUnit XML Report
Maven Surefire
Maven surefire plugin is required to generate the JUnit XML Report.
JUnit 4 Settings
JUnit 5 Settings
If the JUnit 5 features are being utilized, the junit-jupiter-engine needs to be added with additional configurations for fine grained configuration of reports.
Tests need to be annotated with @DisplayName and <usePhrasedXXXName>true</usePhrasedXXXName>
fields need to be set to true as shown below.
For more info, refer Maven Surefire - Using JUnit 5
Ant
If you are using Ant, following tasks need to be used to generate the report.
JUnit 4 - JUnit Task
JUnit 5 - JUnitlauncher Task
Using the junitlauncher
task allows launching the JUnit 5 test launcher and building the test requests to be executed by the test engine(s) supported by JUnit 5. For more information, please refer JUnitLauncher.
Status Mapping JUnit → AIO Tests
JUnit XML | Description | AIO Tests Mapping |
---|---|---|
No tag inside <testcase> means Passed | Passed case | Passed |
</skipped> | Skipped case either by @Ignore or others | Not Run |
</failure> |
| Failed |
</error> |
| Failed |
Mapping automated JUnit tests to AIO Tests
In going with the simplicity advantage of AIO Tests, the JUnit integration has been designed as a simple and non-intrusive integration, without any extra dependency or coding required.
There are 4 ways that a JUnit case is mapped to an AIO Case.
JUnit 5:
@DisplayName
gives a way to specify generic information in the test name instead of the hard rule based Java method names. This can be used to specify an existing AIO Case Key.
For example,@DisplayName("SCRUM-TC-1973 : Test forest gets created.")
Testcase key in Underscores : Since Java and Python, do not allow hyphens in their method signatures, AIO Tests allows case keys with underscores as valid mapping. The AIO Test key can be used in the method name by replacing the hyphen with underscores. eg. if AT-TC-123 is being automated by a test, the test method can be named as
verifyNotNullType_AT_TC_123
and the results of this method will be marked against AT-TC-123.Automation Key: The mapping of an AIO Case to an automated JUnit case can happen through a field Automation Key, which can be specified in the Case in AIO Tests app as shown below. The automation key is the fully qualified name of the test method i.e packagename.classname.methodname.
This can be useful at places where you do not want to add AIO case keys in the name or annotations or where automation is happening first without a manual case being created in AIO Tests.New Case creation: If no case key (e.g. SCRUM-TC-xx) or automation key is found with existing cases, a new case is created with
- title as thename
value from <testcase> tag of the JUnit report
- automation key asclassname.name
from the JUnit report.
- status as Published
- automation status as Automated
- automation owner as user uploading the resultsThe following flow summarizes the logic of JUnit case mapping when a key is not found via @DisplayName and via Automation key
Examples
Below are few examples that show results on testing the ForestCreator.java class.
Mapping using JUnit5 @DisplayName (when settings are done as shown above for DisplayName in SureFire/ANT task)
🖊 Add case key to Display name
@DisplayName("SCRUM-TC-2309 : Verify positive case - all trees get planted") @Test public void testPlantTree() { ForestCreator fc = new ForestCreator(); int plantedTrees = fc.plantTrees(10); Assertions.assertEquals(plantedTrees, 10, "Verify all trees get planted"); } Report: <testcase name="SCRUM-TC-2309 : Verify positive case - all trees get planted" classname="com.aio.tests.junit5.ForestCreatorTests" time="0.001">
Output: On import of the above report in AIO Tests, SCRUM-TC-2309 is updated. If case doesn’t exist then an appropriate error would be thrown
Mapping one automated case with multiple manual cases with JUnit5 @DisplayName
🖊 Add all case keys as part of Display name
@DisplayName("SCRUM-TC-2307, SCRUM-TC-2308,SCRUM-TC-2350 : Verify positive case - all trees get planted") @Test public void testCreateForest() { ForestCreator fc = new ForestCreator(); fc.plantTrees(1); fc.waterThePlants(); fc.removeWeeds(); Assertions.assertEquals(fc.getHealthyTreeCount(), 1, "Verify end to end plant growth"); } Report: <testcase name="SCRUM-TC-2307, SCRUM-TC-2308,SCRUM-TC-2350 : Verify positive case - all trees get planted" classname="com.aio.tests.junit5.ForestCreatorTests" time="0.001">
Output: On import of the above report in AIO Tests, 3 cases are updated with result of the above execution → SCRUM-TC-2307, SCRUM-TC-2308 and SCRUM-TC-2350
. If case doesn’t exist then an appropriate error would be thrown
Mapping JUnit 4 case to existing manual case(s)
🖊 Method name can contain case key with underscores : plantTree_should_returnNumberOfTrees_AT_TC_222.
. OR
🖊 Automation Owner would have to mark the automated key value in an existing case to com.aio.tests.junit4.ForestCreatorTests.plantTree_should_returnNumberOfTrees
🖊 If single automated JUnit case needs to update multiple cases, then the automated key needs to be updated in multiple AIO Tests automation key values
@Test public void plantTree_should_returnNumberOfTrees() { ForestCreator fc = new ForestCreator(); int plantedTrees = fc.plantTrees(10); Assertions.assertEquals(plantedTrees, 10, "Verify all trees get planted"); } Report: <testcase name="plantTree_should_returnNumberOfTrees" classname="com.aio.tests.junit4.ForestCreatorTests" time="0.001">
Output: On import of the above report in AIO Tests, the automation key with com.aio.tests.junit4.ForestCreatorTests.plantTree_should_returnNumberOfTrees
would be searched for and the cases marked with this key would be added to cycle and updated
Parametrized Named Cases
🖊 Add case key as part of the Source and use it in the @ParameterizedTest
name value
Report: <testcase name="Plant Baobab - Case: SCRUM-TC-1621" classname="com.aio.tests.junit5.ForestCreatorTests" time="0.001"> </testcase> <testcase name="Plant Palm - Case: SCRUM-TC-1622" classname="com.aio.tests.junit5.ForestCreatorTests" time="0.001"> </testcase> <testcase name="Plant Banyan - Case: SCRUM-TC-1633" classname="com.aio.tests.junit5.ForestCreatorTests" time="0.001"> </testcase>
Output : On import of the above report in AIO Tests, the 3 data driving the case would update 3 cases with the keys found in the name, in above case
Parametrized Non-Named Cases
🖊 Since there would be no way to differentiate by data, the same case would be updated with multiple runs. AIO Tests discards the value in [ ]
to identify the test name
@RunWith(Parameterized.class) public class ParametrizedRunTest { @Parameterized.Parameters public static Iterable<Object[]> planTrees() { return Arrays.asList(new Object[][] { { "Baobab" }, { "Rainbow Eucalyptus" }, { "Banyan" } }); } Report: <testcase name="plantTree[0]" classname="com.aio.tests.ParametrizedRunNonNamedTest" time="0"/> <testcase name="plantTree[1]" classname="com.aio.tests.ParametrizedRunNonNamedTest" time="0"> <failure message="asdf" type="org.junit.ComparisonFailure"><![CDATA[org.junit.ComparisonFailure: expected:<[BaobabTree]> but was:<[Rainbow Eucalyptus]> at com.aio.tests.ParametrizedRunNonNamedTest.plantTree(ParametrizedRunNonNamedTest.java:33) ]]></failure> </testcase> <testcase name="plantTree[2]" classname="com.aio.tests.ParametrizedRunNonNamedTest" time="0"/>
Output : The case with automation key of com.aio.tests.ParametrizedRunNonNamedTest.plantTree
would be updated - 3 runs would be created with 1st and 3rd run showing pass and 2nd run showing Failed status
Importing Results
Post execution of a JUnit suite, the TEST-<xxx>.xml file can be uploaded either via
AIO Tests REST API call using multipart form-data to upload file
Please follow the above links to continue to import results using either of the options.