AIO Tests supports importing Behave results through it’s support for JUnit reports or results can also be reported using the hooks from the Behave framework and calling AIO Tests REST APIs.
Behave is a tool used for Behaviour driven development (BDD) in Python, that is very similar to Cucumber in Java/Ruby or Specflow for .Net. It uses tests written in a natural language style i.e Gherkin, backed up by Python code. Test scenarios are written in Gherkin “.feature” files. BDD Keyword based steps are glued to a step definition, which is a Python function decorated by a matching string in a step definition module. The feature files act like test scripts.
Behave provides multiple Hooks at various points of execution, which can insert helper logic for test execution. It also provides an out of the box JUnit file, which can simply be generated using a flag.
There are two ways to integrate AIO Tests with Behave either via the JUnit report or via AIO REST API calls in afterScenario hooks of Behave. This document provides an overview on :
Generating the Junit report from Behave tests and uploading it to AIO Tests.
Using AIO Tests REST APIs to report results and much more, using the Behave hooks.
Required Setup
pip install behavepip install requests(Optional, required only if reporting results via APIs)
Reporting results via JUnit file
For the documentation, we are using a feature file from Behave tutorial.
Feature: Showing off AIO Tests with behave
@P0
Scenario: Run a simple passing test
Given we have behave installed
When we implement a test
Then behave will test it for us!
@P1
Scenario: Stronger opponent, which is going to fail
Given the ninja has a third level black-belt
When attacked by Chuck Norris
Then the ninja should run for his life
And fall off a cliff
Running your cases and generating a Junit XML Report file
behave provides 2 different concepts for reporting results of a test run:
reporters
formatters
The JUnit reporter provides JUnit XML-like output and can simply be generated by adding a junit flag from the behave CLI as below :
behave --junit
This generates one xml for each feature file that has been selected for running:
Mapping Cases with AIO Tests
AIO Tests supports reporting results against existing cases in AIO Tests or creating tests using the reports.
AIO Tests generates a unique key for each test created in AIO Test, example AT-TC-1. To map an automated case to an existing AIO Test, the test name can take the testcase key generated in its description. e.g.AT-TC-299 : Login with existing user
In the example below,
for scenario 1, Behave test/scenario maps to one AIO Tests
for scenario 2, a single Behave test/scenario maps to two manual cases in AIO Tests.
Feature: Showing off AIO Tests with behave
@P0
Scenario: AT-TC-299 - Run a simple passing test
Given we have behave installed
When we implement a test
Then behave will test it for us!
@P1
Scenario: AT-TC-300, AT-TC-301 - Stronger opponent, which is going to fail
Given the ninja has a third level black-belt
When attacked by Chuck Norris
Then the ninja should run for his life
And fall off a cliff
Running the above creates the following JUnit XML. Note that the key becomes part of the name in the report
<testsuite name="behaveDemo.Showing off AIO Tests with behave" tests="2" errors="0" failures="1" skipped="0" time="0.003063" timestamp="2022-07-26T12:50:11.806647" hostname="-MacBook-Pro.local">
<testcase classname="behaveDemo.Showing off AIO Tests with behave" name="AT-TC-299 Run a simple passing test" status="passed" time="0.000272"><system-out>
<![CDATA[
@scenario.begin
...
>
</system-out></testcase>
<testcase classname="behaveDemo.Showing off AIO Tests with behave" name="AT-TC-300, AT-TC-301 - Stronger opponent, which is going to fail" status="failed" time="0.002791"><failure type="AssertionError" message="">
<![CDATA[
Failing step: And fall off a cliff ... failed in 0.002s
Location: features/behaveDemo.feature:16
Traceback (most recent call last):
File "/Users/varshneyn/PycharmProjects/pythonProject/venv/lib/python3.8/site-packages/behave/model.py", line 1329, in run
match.run(runner.context)
File "/Users/varshneyn/PycharmProjects/pythonProject/venv/lib/python3.8/site-packages/behave/matchers.py", line 98, in run
self.func(context, *args, **kwargs)
File "features/steps/tutorial.py", line 29, in step_impl
assert True is not True
AssertionError
]]>
</failure><system-out>
<![CDATA[
@scenario.begin
...
@scenario.end
--------------------------------------------------------------------------------
]]>
</system-out></testcase></testsuite>
Uploading results to AIO Tests
Post execution of a suite, the TEST-<xxx>.xml file can be uploaded either via
AIO Tests REST API call using multipart form-data to upload file
Please follow the above links to continue to import results using either of the options.
Uploading the above file for the first time will
create new cases in the system if there is no key found in the classname.name. The new case is created with
- title as thenamevalue from <testcase> tag of the JUnit report
- automation key asclassname.namefrom the JUnit report.
- status as Published
- automation status as Automated
- automation owner as user uploading the resultsAdd the newly created case to the cycle being uploaded to
Mark the details of the run
Execution mode is set to Automated
Duration of run is set to Actual Effort
Status of run is set based on status mapping table below
Failures and errors are reported as Run Level comments
If the same file is uploaded again, the cases will be identified using the automation key (
classname.name )and would be updated, instead of creating new cases.
Status Mapping JUnit → AIO Tests
JUnit XML | Description | AIO Tests Mapping |
|---|---|---|
No tag inside <testcase> means Passed | Passed case | Passed |
</skipped> | Skipped case either by @Ignore or others | Not Run |
</failure> |
| Failed |
</error> |
| Failed |
Reporting results via Behave Hooks and AIO Tests REST APIs
AIO Tests provides a rich set of APIs for Execution Management, using which users can not only report execution status, but also add effort, actual results, comments, defects and attachments to runs as well as steps.
AIO Tests also provides APIs to create cycles and to add cases to cycles for execution planning.
The basic sample below will show how Behave Hooks can leverage the AIO Tests REST APIs to report results. In the environment.py the before_scenario and after_scenario methods can be used to make AIO API calls.
Establish a convention for AIO Tests Case keys
Any convention can be established and the code consuming it can cater to the convention. The below example demonstrates this by adding tags made up of case keys
@AIOTestsDemo
Feature: Showing off AIO Tests with behave
@P0 @AT-TC-299
Scenario: Run a simple passing test
Given we have behave installed
When we implement a test
Then behave will test it for us!
Behave environment.py
In the before_scenario hook, the tag can be read and can be added to a cycle. In the after_scenario hook, the details of the case can be added to the run returned in the before_scenario
The below is a basic example of what can be done with the hooks and AIO Tests APIs. It is recommended to add appropriate error handling and enhance it based on your automation requirements.
import requests;
import json;
projectKey = "NVTES" #put in your Jira project key
cycleKey = "NVTES-CY-172" #cycle can be created in before_all using APIs as well
headers = {"Authorization": "AIO Tests Authorization", 'Accept': 'application/json', "Content-type": "application/json"};
def findAioKey(tags):
if tags is not None:
result = [i for i in tags if i.startswith('NVTES-TC-')]
return result
return None
def before_scenario(context, scenario):
aioKey = findAioKey(scenario.tags)
if aioKey is not None and len(aioKey) > 0:
print("AIO Key found " + aioKey[0]);
scenario.aiokey = aioKey[0]
scenario.testrunid = add_run(projectKey, cycleKey, aioKey[0])
def after_scenario(context, scenario):
if hasattr(scenario, 'aiokey'):
add_results(projectKey, cycleKey, scenario.testrunid, status);
def add_results(projectKey, cycleKey, testRunId, caseStatus):
data = {
"testRunStatus": {
"name": caseStatus
},
"effort": 600,
"comments": [{"comment": "some comment"}]
}
response = requests.post(
_url('/project/{}/testcycle/{}/testrun/{}'.format(projectKey, cycleKey, testRunId)), json=data, headers=headers)
print(response.text)
def add_run(projectKey, cycleKey, testKey):
response = requests.post(_url('/project/{}/testcycle/{}/testcase/{}'.format(projectKey, cycleKey, testKey)), headers=headers)
print(response.text)
testRunId = json.loads(response.text)["ID"]
print(testRunId)
return (testRunId)
def _url(path):
base_url = "https://tcms.aiojiraapps.com/aio-tcms/api/v1" #Use Jira server url if on Jira server
return base_url + path
In the example above, the add_run method uses requests to make an HTTP call.
It uses the scenario tags to identify the case key [ based on the convention we established]
Create a POST request
URL : For cloud the url host would be
https://tcms.aioreports.com/aio-tcms/api/v1. For Jira server, it would be the native Jira server hostname.Authorization : Please refer to Rest API Authentication to understand how to authorize users. The authentication information goes in the
headers: {'Authorization': '<Auth based on Jira Cloud/Jira Server>'},POST Body : The body consists of data from the test object.
If required, the basic example can be extended to upload attachments against the case using the upload attachment API.