Postman via Newman Report

Owned by APPWRK Writer
Last updated: Mar 18, 2024 by Niharika [AIO Tests]
6 min read

AIO Tests provides integration with Postman tests by allowing users to import their Newman-generated JSON report via REST calls or with a custom Newman Reporter. Results can be updated for existing AIO Tests Cases by mapping automated cases or by creating new ones from the Newman results JSON file. 

Having the automated results reporting in AIO Tests can help in having a holistic view of the coverage of requirements (unit to manual to functional and end-to-end automated tests).

This article discusses how are API tests created and the results of the tests uploaded to AIO Tests using Newman results.

In this documentation, you’ll understand:

Postman and Newman

Postman is an API platform for building and using APIs. It is also an API client that makes it easy for developers to design, mock, create, share, test and document APIs. With Postman, you can send requests to connect to APIs you are working with. Requests can retrieve, add, delete, and update data. Requests can send parameters, authorization details, and any data that is required.

Newman is an open-source command-line Collection Runner for Postman. It enables you to run and test a Postman Collection directly from the command line, making it possible to run API tests in continuous integration servers and build systems.

Newman Setup

  1. To run Newman, ensure that you have Node.js >= v10. Install Node.js via package manager.

  2. npm install -g newman or brew install newman

Execute Postman Collection via Newman

Postman requests and examples can be grouped into collections to keep a workspace organized, collaborate with teammates, generate API documentation and API tests, and automate request runs.

  1. Create a Postman Collection. For details on creating Postman collections, please refer to Creating Collections and Adding Requests to Collections.

  2. Export the collection (Steps to Export). Exporting creates a collection.json file.

  3. Using the newman CLI, run the collection
    newman run sample-collection.json

  4. Environments provide a set of variables that one can use within collections. If an environment file is required, then export the environment file and use it with Newman run as follows:
    newman run mycollection.json -e env.json

Sample Collection

Generating JSON Report

The JSON reporter comes out of the box with Newman. To generate a JSON report for a newman run, provide --reporters json as a CLI option.

It takes the path to the file, where to write the report --reporter-json-export <path>. If not specified, the file will be written to newman/ in the current working directory.

So, a complete command to run and generate a JSON report will be as below:
newman run mycollection.json -e env.json --reporters json --reporter-json-export <path>

Sample Report

The JSON report is a comprehensive output of the run summary. It contains the details of all requests, their tests, the responses and much more. A sample report is as below:

Integration with AIO Tests

AIO Tests supports Postman API Tests via the BDD/Gherkin format of cases. Using the Newman output of runs, AIO Tests allows users to:

  • either report results against existing cases in AIO Tests

  • or create new cases on the fly.

A sample API case in AIO Tests looks like below. The next section shows how Postman data of requests, tests and assertions are mapped to an AIO Test.

image-20240223-055040.png

Field Mapping (Postman → AIO Tests)

The below table summarizes how a Postman API Tests gets translated to an AIO Case.

Postman field

AIO Tests Field

Example

Postman field

AIO Tests Field

Example

Name of request

Title

PublicApi - Get TestCase Priority

Name of request

Automation key

PublicApi - Get TestCase Priority

Request endpoint

First step in Given

GET {{SERVER_URL}}/project/{{JIRA_PROJECT_ID}}/config/testcase/priority

Query Params

Data in first step

Query: startAt=0&maxResults=100&needDataInRTF=true

Body params

Data in first step

{"testRunStatus": "Passed<b>NR</b>1", "effort": 180, "comments": ["Comment line1", "Comment line2"] }

Form Data

Data in first step [FormData:
key1 = value1

key2=value2]

FormData: file = "/file.json"
cleanResults = true
fetchDetails = true

URL Encoded Body

Data in first step [ URLEncoded: key = value ]

URLEncoded: what = for

Headers

No mapping

 

First assertion (if any)

Step 2 with Then keyword

THEN Response must be valid and have a body

Rest of the assertions

Step 3 and forth with And as keyword

AND Validate Priorities in Response

Reponse size, response code, response time

Actual result of first step

 

Variable values [ Env, collection ]

Actual result of first step

 

Request Error [ happens when server URL is invalid]

Actual result - against first step

RequestError{ errno=-3008, code='ENOTFOUND', syscall='getaddrinfo', hostname='{{cloud url}}'}

Assertion Failures

Actual result

AssertionError: expected response to have status reason 'OK' but got 'UNDEFINED' at Object.eval sandbox-script.js:1:1)

Mapping Postman Tests to Existing AIO Tests

If cases have been created in AIO Tests to uniquely map the AIO Case to the API Test, the case key can be added to the name of the API, as shown below:

Adding case key to API name is not mandatory. Even without a case key, the case would be identified using the automation key. However, in the event, where the API name changes, AIO would recognize it as a new case and create a new one. To avoid this situation, it is recommended to add case key, so that future changes to the name of the API still get written to the same case.

Creating Postman Tests in AIO Tests

If no case key is added to the name of the API, then a new case is created in AIO Tests, with the field mappings as explained in the section: Field mapping (Postman → AIO Tests)

The automation key of this newly created case is set to the name of the API. The subsequent times when the API test is run, the cases are identified by the automation key.

In the above example, the first 2 cases will be reported to AT-TC-223 and AT-TC-224. However, the 3rd and the 4th requests will be created as new cases. Below is an example of a newly created case:

Reporting Results

To report the results of a collection run AIO Tests provide 2 ways.

  1. The newman-reporter-aiotests allows users to set up an AIO Tests config in their Newman run if running Newman as a library or via the command line. The reporter gives additional capabilities to create cycles for each run or to reuse existing cycles.

  2. Users may choose to import the JSON report (generated by the inbuilt JSON reporter), via the Import Result API or through the Cycle Import Results option or through the Jenkins plugin. To read more about these options, please refer to Importing Results in AIO Tests.

newman-reporter-aiotests

AIO Tests has introduced a custom reporter for Newman. The newman-reporter-aiotests simplifies reporting results from Newman with an easy setup.

Please refer to Newman AIO Tests Reporter for instructions on installation and usage.

For further queries and suggestions, feel free to reach out to our customer support via help@aiotests.com.