Postman via Newman Report
AIO Tests provides integration with Postman tests by allowing users to import their Newman-generated JSON report via REST calls or with a custom Newman Reporter. Results can be updated for existing AIO Tests Cases by mapping automated cases or by creating new ones from the Newman results JSON file.
Having the automated results reporting in AIO Tests can help in having a holistic view of the coverage of requirements (unit to manual to functional and end-to-end automated tests).
This article discusses how are API tests created and the results of the tests uploaded to AIO Tests using Newman results.
In this documentation, you’ll understand:
Postman and Newman
Postman is an API platform for building and using APIs. It is also an API client that makes it easy for developers to design, mock, create, share, test and document APIs. With Postman, you can send requests to connect to APIs you are working with. Requests can retrieve, add, delete, and update data. Requests can send parameters, authorization details, and any data that is required.
Newman is an open-source command-line Collection Runner for Postman. It enables you to run and test a Postman Collection directly from the command line, making it possible to run API tests in continuous integration servers and build systems.
Newman Setup
To run Newman, ensure that you have Node.js >= v10. Install Node.js via package manager.
npm install -g newman
orbrew install newman
Execute Postman Collection via Newman
Postman requests and examples can be grouped into collections to keep a workspace organized, collaborate with teammates, generate API documentation and API tests, and automate request runs.
Create a Postman Collection. For details on creating Postman collections, please refer to Creating Collections and Adding Requests to Collections.
Export the collection (Steps to Export). Exporting creates a collection.json file.
Using the newman CLI, run the collection
newman run sample-collection.json
Environments provide a set of variables that one can use within collections. If an environment file is required, then export the environment file and use it with Newman run as follows:
newman run mycollection.json -e env.json
Sample Collection
Generating JSON Report
The JSON reporter comes out of the box with Newman. To generate a JSON report for a newman run, provide --reporters json
as a CLI option.
It takes the path to the file, where to write the report --reporter-json-export <path>
. If not specified, the file will be written to newman/
in the current working directory.
So, a complete command to run and generate a JSON report will be as below:newman run mycollection.json -e env.json --reporters json --reporter-json-export <path>
Sample Report
The JSON report is a comprehensive output of the run summary. It contains the details of all requests, their tests, the responses and much more. A sample report is as below:
Integration with AIO Tests
AIO Tests supports Postman API Tests via the BDD/Gherkin format of cases. Using the Newman output of runs, AIO Tests allows users to:
either report results against existing cases in AIO Tests
or create new cases on the fly.
A sample API case in AIO Tests looks like below. The next section shows how Postman data of requests, tests and assertions are mapped to an AIO Test.
Field Mapping (Postman → AIO Tests)
The below table summarizes how a Postman API Tests gets translated to an AIO Case.
Postman field | AIO Tests Field | Example |
---|---|---|
Name of request | Title | PublicApi - Get TestCase Priority |
Name of request | Automation key | PublicApi - Get TestCase Priority |
Request endpoint | First step in Given | GET {{SERVER_URL}}/project/{{JIRA_PROJECT_ID}}/config/testcase/priority |
Query Params | Data in first step | Query: startAt=0&maxResults=100&needDataInRTF=true |
Body params | Data in first step | {"testRunStatus": "Passed<b>NR</b>1", "effort": 180, "comments": ["Comment line1", "Comment line2"] } |
Form Data | Data in first step [FormData: key2=value2] | FormData: file = "/file.json" |
URL Encoded Body | Data in first step [ URLEncoded: key = value ] | URLEncoded: what = for |
Headers | No mapping |
|
First assertion (if any) | Step 2 with Then keyword |
|
Rest of the assertions | Step 3 and forth with And as keyword |
|
Reponse size, response code, response time | Actual result of first step |
|
Variable values [ Env, collection ] | Actual result of first step |
|
Request Error [ happens when server URL is invalid] | Actual result - against first step | RequestError{ errno=-3008, code='ENOTFOUND', syscall='getaddrinfo', hostname='{{cloud url}}'} |
Assertion Failures | Actual result |
|
Mapping Postman Tests to Existing AIO Tests
If cases have been created in AIO Tests to uniquely map the AIO Case to the API Test, the case key can be added to the name of the API, as shown below:
Adding case key to API name is not mandatory. Even without a case key, the case would be identified using the automation key. However, in the event, where the API name changes, AIO would recognize it as a new case and create a new one. To avoid this situation, it is recommended to add case key, so that future changes to the name of the API still get written to the same case.
Creating Postman Tests in AIO Tests
If no case key is added to the name of the API, then a new case is created in AIO Tests, with the field mappings as explained in the section: Field mapping (Postman → AIO Tests)
The automation key of this newly created case is set to the name of the API. The subsequent times when the API test is run, the cases are identified by the automation key.
In the above example, the first 2 cases will be reported to AT-TC-223 and AT-TC-224. However, the 3rd and the 4th requests will be created as new cases. Below is an example of a newly created case:
Reporting Results
To report the results of a collection run AIO Tests provide 2 ways.
The newman-reporter-aiotests allows users to set up an AIO Tests config in their Newman run if running Newman as a library or via the command line. The reporter gives additional capabilities to create cycles for each run or to reuse existing cycles.
Users may choose to import the JSON report (generated by the inbuilt JSON reporter), via the Import Result API or through the Cycle Import Results option or through the Jenkins plugin. To read more about these options, please refer to Importing Results in AIO Tests.
newman-reporter-aiotests
AIO Tests has introduced a custom reporter for Newman. The newman-reporter-aiotests simplifies reporting results from Newman with an easy setup.
Please refer to Newman AIO Tests Reporter for instructions on installation and usage.
For further queries and suggestions, feel free to reach out to our customer support via help@aiotests.com.