{"type":"doc","content":[{"type":"paragraph","content":[{"text":"AIO Tests supports importing Behave results through its support for JUnit reports or results can also be reported using the hooks from the Behave framework and calling AIO Tests REST APIs.","type":"text"}]},{"type":"paragraph","content":[{"text":"Behave","type":"text","marks":[{"type":"em"},{"type":"link","attrs":{"href":"https://behave.readthedocs.io/en/stable/"}}]},{"text":" is a tool used for Behaviour-driven development (","type":"text"},{"text":"BDD","type":"text","marks":[{"type":"em"}]},{"text":") in ","type":"text"},{"text":"Python, ","type":"text","marks":[{"type":"em"}]},{"text":"that is very similar to ","type":"text"},{"text":"Cucumber","type":"text","marks":[{"type":"link","attrs":{"href":"https://cucumber.io/"}}]},{"text":" in Java/Ruby or ","type":"text"},{"text":"Specflow","type":"text","marks":[{"type":"link","attrs":{"href":"https://specflow.org/"}}]},{"text":" for .Net. It uses tests written in a natural language style i.e. Gherkin, backed up by Python code. Test scenarios are written in ","type":"text"},{"text":"Gherkin “.feature” files","type":"text","marks":[{"type":"link","attrs":{"href":"https://behave.readthedocs.io/en/latest/gherkin/#gherkin-feature-testing-language"}}]},{"text":". BDD keyword-based steps are glued to a ","type":"text"},{"text":"step definition","type":"text","marks":[{"type":"link","attrs":{"href":"https://behave.readthedocs.io/en/latest/api/#step-functions"}}]},{"text":", which is a Python function decorated by a matching string in a step definition module. The feature files act like test scripts. ","type":"text"},{"type":"hardBreak"},{"text":"Behave provides multiple ","type":"text"},{"text":"Hooks","type":"text","marks":[{"type":"link","attrs":{"href":"https://behave.readthedocs.io/en/latest/api/#environment-file-functions"}}]},{"text":" at various points of execution, which can insert helper logic for test execution. It also provides an out-of-the-box ","type":"text"},{"text":"JUnit","type":"text","marks":[{"type":"link","attrs":{"href":"https://behave.readthedocs.io/en/stable/formatters.html?highlight=junit#reporters"}}]},{"text":" file, which can simply be generated using a flag.","type":"text"}]},{"type":"paragraph","content":[{"text":"There are two ways to integrate AIO Tests with Behave either via the JUnit report or via AIO REST API calls in afterScenario hooks of Behave. This document provides an overview on:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Generating the Junit report from Behave tests and uploading it to AIO Tests.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Using ","type":"text"},{"text":"AIO Tests REST APIs","type":"text","marks":[{"type":"link","attrs":{"href":"https://tcms.aiojiraapps.com/aio-tcms/aiotcms-static/api-docs/"}}]},{"text":" to report results and much more, using the Behave hooks.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"In this documentation, you’ll understand:","type":"text"}]},{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"toc","parameters":{"macroParams":{"minLevel":{"value":"1"},"maxLevel":{"value":"6"},"outline":{"value":"false"},"style":{"value":"none"},"type":{"value":"list"},"printable":{"value":"false"}},"macroMetadata":{"macroId":{"value":"6912df9a-f226-4b12-b4b6-aa3ab2875a6a"},"schemaVersion":{"value":"1"},"title":"Table of Contents"}},"localId":"39493d87-cfbb-496c-a881-378855622f98"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Required Setup","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"pip install behave","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"pip install requests","type":"text","marks":[{"type":"code"}]},{"text":" (Optional, required only if reporting results via APIs)","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Reporting Results via JUnit File","type":"text"}]},{"type":"paragraph","content":[{"text":"For the documentation, we are using a feature file from the ","type":"text"},{"text":"Behave tutorial","type":"text","marks":[{"type":"link","attrs":{"href":"https://behave.readthedocs.io/en/stable/tutorial.html"}}]},{"text":".","type":"text"}]},{"type":"codeBlock","content":[{"text":"Feature: Showing off AIO Tests with behave\n\n @P0\n Scenario: Run a simple passing test\n Given we have behave installed\n When we implement a test\n Then behave will test it for us!\n\n @P1\n Scenario: Stronger opponent, which is going to fail\n Given the ninja has a third level black-belt\n When attacked by Chuck Norris\n Then the ninja should run for his life\n And fall off a cliff","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Running your Cases and Generating a Junit XML Report File","type":"text"}]},{"type":"paragraph","content":[{"text":"Behave","type":"text","marks":[{"type":"link","attrs":{"href":"https://pypi.python.org/pypi/behave"}}]},{"text":" provides 2 different concepts for reporting the results of a test run:","type":"text"}]},{"type":"blockquote","content":[{"type":"paragraph","content":[{"text":"reporters","type":"text"}]},{"type":"paragraph","content":[{"text":"formatters","type":"text"}]}]},{"type":"paragraph","content":[{"text":"The JUnit reporter provides JUnit XML-like output and can simply be generated by adding a JUnit flag from the behave CLI as below :","type":"text"}]},{"type":"paragraph","content":[{"text":"behave --junit","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"This generates one XML for each feature file that has been selected for running:","type":"text"}]},{"type":"expand","attrs":{"title":"Sample result file"},"content":[{"type":"codeBlock","content":[{"text":"\n\n\n\n\n\n\n\n\n\n\n","type":"text"}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Mapping Cases with AIO Tests","type":"text"}]},{"type":"paragraph","content":[{"text":"AIO Tests supports reporting results against existing cases in AIO Tests or creating tests using the reports.","type":"text"}]},{"type":"paragraph","content":[{"text":"AIO Tests generates a unique key for each test created in AIO Test, for example, AT-TC-1. To map an automated case to an existing AIO Test, ","type":"text"},{"text":"the test name can take the test case key generated in its description. e.g.","type":"text","marks":[{"type":"strong"},{"type":"em"}]},{"text":"AT-TC-299: Login with an existing user","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"In the example below,","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"For scenario 1, the Behave test/scenario maps to one AIO Tests.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"For scenario 2, a single Behave test/scenario maps to two manual cases in AIO Tests.","type":"text"}]}]}]},{"type":"codeBlock","content":[{"text":"Feature: Showing off AIO Tests with behave\n\n @P0\n Scenario: AT-TC-299 - Run a simple passing test\n Given we have behave installed\n When we implement a test\n Then behave will test it for us!\n\n\n @P1\n Scenario: AT-TC-300, AT-TC-301 - Stronger opponent, which is going to fail\n Given the ninja has a third level black-belt\n When attacked by Chuck Norris\n Then the ninja should run for his life\n And fall off a cliff","type":"text"}]},{"type":"paragraph","content":[{"text":"Running the above creates the following JUnit XML. Note that the key becomes part of the name in the report.","type":"text"}]},{"type":"codeBlock","content":[{"text":"\n

\n\n

\n\n\n

\n\n

","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Uploading Results to AIO Tests","type":"text"}]},{"type":"paragraph","content":[{"text":"Post execution of a suite, the TEST-.xml file can be uploaded either via:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"AIO Tests REST API ","type":"text","marks":[{"type":"link","attrs":{"href":"/wiki/spaces/AioTests/pages/2026569775"}}]},{"text":"calls using multipart form data to upload file","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Cycle options in the Cycles Tab inside AIO Tests","type":"text","marks":[{"type":"link","attrs":{"href":"/wiki/spaces/AioTests/pages/2026700831"}}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"via ","type":"text"},{"text":"AIO Tests Jenkins Plugin","type":"text","marks":[{"type":"link","attrs":{"href":"https://plugins.jenkins.io/aio-tests/"}}]}]}]}]},{"type":"paragraph","content":[{"text":"Please follow the above links to continue to import results using either of the options.","type":"text"}]},{"type":"paragraph","content":[{"text":"Uploading the above file for the first time will","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Create new cases in the system if there is no key found in the ","type":"text"},{"text":"classname.name","type":"text","marks":[{"type":"link","attrs":{"href":"http://classname.name"}}]},{"text":". The new case is created with:","type":"text"},{"type":"hardBreak"},{"text":"- Title as the ","type":"text"},{"text":"name","type":"text","marks":[{"type":"code"}]},{"text":" value from tag of the JUnit report","type":"text"},{"type":"hardBreak"},{"text":"- Automation key as ","type":"text"},{"text":"classname.name","type":"text","marks":[{"type":"code"}]},{"text":" from the JUnit report. ","type":"text"},{"type":"hardBreak"},{"text":"- Status as Published","type":"text"},{"type":"hardBreak"},{"text":"- Automation status as Automated","type":"text"},{"type":"hardBreak"},{"text":"- Automation owner as a user uploading the results","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Add the newly created case to the cycle being uploaded.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Mark the details of the run.","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Execution mode is set to Automated.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The duration of a run is set to the Actual Effort.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The status of a run is set based on the status mapping table below.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Failures and errors are reported as Run Level comments.","type":"text"}]}]}]}]}]},{"type":"blockquote","content":[{"type":"paragraph","content":[{"text":"If the same file is uploaded again, the cases will be identified using the automation key (","type":"text"},{"text":"classname.name )","type":"text","marks":[{"type":"code"}]},{"text":"and would be updated, instead of creating new cases.","type":"text"}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Status Mapping JUnit → AIO Tests","type":"text"}]},{"type":"table","attrs":{"layout":"default","width":760.0,"localId":"77c3647f-c24f-4638-91c1-84a0d15d0aa7"},"content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"background":"var(--ds-background-accent-gray-subtlest, #F4F5F7)","rowspan":1,"colwidth":[157.0]},"content":[{"type":"paragraph","content":[{"text":"JUnit XML","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableHeader","attrs":{"colspan":1,"background":"var(--ds-background-accent-gray-subtlest, #F4F5F7)","rowspan":1,"colwidth":[350.0]},"content":[{"type":"paragraph","content":[{"text":"Description","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableHeader","attrs":{"colspan":1,"background":"var(--ds-background-accent-gray-subtlest, #F4F5F7)","rowspan":1,"colwidth":[253.0]},"content":[{"type":"paragraph","content":[{"text":"AIO Tests Mapping","type":"text","marks":[{"type":"strong"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[157.0]},"content":[{"type":"paragraph","content":[{"text":"No tag inside means Passed","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[350.0]},"content":[{"type":"paragraph","content":[{"text":"Passed case","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[253.0]},"content":[{"type":"paragraph","content":[{"text":"Passed","type":"text"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[157.0]},"content":[{"type":"paragraph","content":[{"text":"","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[350.0]},"content":[{"type":"paragraph","content":[{"text":"Skipped case either by @Ignore or others","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[253.0]},"content":[{"type":"paragraph","content":[{"text":"Not Run","type":"text"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[157.0]},"content":[{"type":"paragraph","content":[{"text":"","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[350.0]},"content":[{"type":"paragraph","content":[{"text":"Indicates that the test failed. A failure is a test which the code has explicitly failed by using the mechanisms for that purpose. e.g., via an assertEquals.","type":"text","marks":[{"type":"code"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[253.0]},"content":[{"type":"paragraph","content":[{"text":"Failed","type":"text"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[157.0]},"content":[{"type":"paragraph","content":[{"text":"","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[350.0]},"content":[{"type":"paragraph","content":[{"text":"Indicates that the test errored. An errored test is one that had an unanticipated problem. e.g., an unchecked throwable;","type":"text","marks":[{"type":"code"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[253.0]},"content":[{"type":"paragraph","content":[{"text":"Failed","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Reporting Results via Behave Hooks and AIO Tests REST APIs","type":"text"}]},{"type":"paragraph","content":[{"text":"AIO Tests provides a rich set of APIs for ","type":"text"},{"text":"Execution Management","type":"text","marks":[{"type":"link","attrs":{"href":"https://tcms.aiojiraapps.com/aio-tcms/aiotcms-static/api-docs/#/Execution%20management"}}]},{"text":", using which users can not only report execution status, but also add effort, actual results, comments, defects and attachments to runs as well as steps. ","type":"text"},{"type":"hardBreak"},{"text":"AIO Tests also provides APIs to ","type":"text"},{"text":"create cycles","type":"text","marks":[{"type":"link","attrs":{"href":"https://tcms.aiojiraapps.com/aio-tcms/aiotcms-static/api-docs/#/Cycle%20Operations"}}]},{"text":" and to ","type":"text"},{"text":"add cases to cycles","type":"text","marks":[{"type":"link","attrs":{"href":"https://tcms.aiojiraapps.com/aio-tcms/aiotcms-static/api-docs/#/Execution%20management/addCaseToCycle"}}]},{"text":" for execution planning.","type":"text"}]},{"type":"paragraph","content":[{"text":"The basic sample below will show how Behave Hooks can leverage the AIO Tests REST APIs to report results. In the ","type":"text"},{"text":"environment.py","type":"text","marks":[{"type":"code"}]},{"text":" the ","type":"text"},{"text":"before_scenario","type":"text","marks":[{"type":"code"}]},{"text":" and ","type":"text"},{"text":"after_scenario","type":"text","marks":[{"type":"code"}]},{"text":" methods can be used to make AIO API calls.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Establish a Convention for AIO Tests Case Keys","type":"text"}]},{"type":"paragraph","content":[{"text":"Any convention can be established and the code consuming it can cater to the convention. The below example demonstrates this by adding tags made up of case keys.","type":"text"}]},{"type":"codeBlock","content":[{"text":"@AIOTestsDemo\nFeature: Showing off AIO Tests with behave\n\n @P0 @AT-TC-299\n Scenario: Run a simple passing test\n Given we have behave installed\n When we implement a test\n Then behave will test it for us!","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Behave environment.py","type":"text"}]},{"type":"paragraph","content":[{"text":"In the ","type":"text"},{"text":"before_scenario","type":"text","marks":[{"type":"code"}]},{"text":" hook, the tag can be read and can be added to a cycle. In the ","type":"text"},{"text":"after_scenario","type":"text","marks":[{"type":"code"}]},{"text":" hook, the details of the case can be added to the run returned in the ","type":"text"},{"text":"before_scenario","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"The below is a basic example of what can be done with the hooks and AIO Tests APIs. It is recommended to add appropriate error handling and enhance it based on your automation requirements.","type":"text","marks":[{"type":"em"}]}]}]},{"type":"codeBlock","content":[{"text":"import requests;\nimport json;\n\nprojectKey = \"NVTES\" #put in your Jira project key\ncycleKey = \"NVTES-CY-172\" #cycle can be created in before_all using APIs as well\nheaders = {\"Authorization\": \"AIO Tests Authorization\", 'Accept': 'application/json', \"Content-type\": \"application/json\"};\n\ndef findAioKey(tags):\n if tags is not None:\n result = [i for i in tags if i.startswith('NVTES-TC-')]\n return result\n return None\n \ndef before_scenario(context, scenario):\n aioKey = findAioKey(scenario.tags)\n if aioKey is not None and len(aioKey) > 0:\n print(\"AIO Key found \" + aioKey[0]);\n scenario.aiokey = aioKey[0]\n scenario.testrunid = add_run(projectKey, cycleKey, aioKey[0])\n\ndef after_scenario(context, scenario):\n if hasattr(scenario, 'aiokey'):\n add_results(projectKey, cycleKey, scenario.testrunid, status);\n\ndef add_results(projectKey, cycleKey, testRunId, caseStatus):\n\n data = {\n \"testRunStatus\": {\n \"name\": caseStatus\n },\n \"effort\": 600,\n \"comments\": [{\"comment\": \"some comment\"}]\n }\n response = requests.post(\n _url('/project/{}/testcycle/{}/testrun/{}'.format(projectKey, cycleKey, testRunId)), json=data, headers=headers)\n print(response.text)\n\ndef add_run(projectKey, cycleKey, testKey):\n response = requests.post(_url('/project/{}/testcycle/{}/testcase/{}'.format(projectKey, cycleKey, testKey)), headers=headers)\n print(response.text)\n testRunId = json.loads(response.text)[\"ID\"]\n print(testRunId)\n return (testRunId)\n\ndef _url(path):\n base_url = \"https://tcms.aiojiraapps.com/aio-tcms/api/v1\" #Use Jira server url if on Jira server\n return base_url + path","type":"text"}]},{"type":"paragraph","content":[{"text":"In the example above, the ","type":"text"},{"text":"add_run","type":"text","marks":[{"type":"code"}]},{"text":" method uses requests to make an HTTP call.","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"It uses the scenario tags to identify the case key [ based on the convention we established]","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Create a POST request.","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"URL:","type":"text","marks":[{"type":"strong"}]},{"text":" For cloud the URL host would be ","type":"text"},{"text":"https://tcms.aiojiraapps.com/aio-tcms/api/v1","type":"text","marks":[{"type":"code"}]},{"text":". For the Jira server, it would be the native Jira server hostname.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Authorization","type":"text","marks":[{"type":"strong"}]},{"text":": Please refer to ","type":"text"},{"text":"Rest API Authentication","type":"text","marks":[{"type":"link","attrs":{"href":"/wiki/spaces/AioTests/pages/2025521186"}}]},{"text":" to understand how to authorize users. The authentication information goes in the ","type":"text"},{"text":"headers: {'Authorization': ''},","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"POST Body: ","type":"text","marks":[{"type":"strong"}]},{"text":"The body consists of data from the test object.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"If required, the basic example can be extended to upload attachments against the case using the ","type":"text"},{"text":"upload attachment API.","type":"text","marks":[{"type":"link","attrs":{"href":"https://tcms.aiojiraapps.com/aio-tcms/aiotcms-static/api-docs/#/Execution%20management/addAttachmentsToLatestRun"}}]}]}]}]}]}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"For further queries and suggestions, feel free to reach out to our customer support via ","type":"text","marks":[{"type":"strong"}]},{"text":"help@aiotests.com","type":"text","marks":[{"type":"underline"},{"type":"strong"},{"type":"link","attrs":{"href":"mailto:help@aiotests.com"}}]},{"text":".","type":"text","marks":[{"type":"strong"}]}]}],"version":1}

Browser not supported