Test Automation

This blueprint focuses just on the test automation portion of the API lifecycle, highlighting the different types of tests that can be run, how monitors can be used to automate those tests on a schedule from multiple regions, and then how to tap into the observability opportunities that exist when it comes to test automation. Test automation should provide you with a 50K foot view of how testing works in conjunction with monitors, and what observability is available by default, published using visualizer, or introduced through integrations with external solutions, providing many different ways of being able to see and understand test automation for API operations.

Define

The most important first step of any API lifecycle is to make sure the operations around an API are properly defined, laying the foundation for being able to effectively design and bring an API to life, while also establishing a known place, or places to go to get all the information you need regarding each individual API, or groups of APIs. A little planning and organization at this early step of the API journey can go a long way towards ensuring the overall health and velocity of an API, and the applications and integrations that will depend on each internal, partner, or public API being delivered.

Team Workspace

Any API you are looking to develop should have a known location where any stakeholder can get up to speed on what is happening with the development and operation of an API. Postman team workspaces provide a single location to publish APIs, documentation, mock servers, tests, and other artifacts, providing a known location that is discoverable across teams. Establishing early on a place for teams to engage around the design, development, and operation of an API, that makes the API lifecycle observable by default. Team workspaces are only accessible internally to your team members who have an account under your designated organizational Postman team, making it available across search, and via workspace browsing for your team to discover and begin collaborating and working across APIs being developed within each workspace.

Team Members

Once a team or public workspace has been established for an API, or group of APIs, it is time to define and invite all relevant team members to the workspace. Sitting down to make sure your team is well defined and invited to your workspace and repositories allows you to think through who is involved, what roles they have in moving an API forward, and is something you can revisit on a regular basis. It is also something that allows others to easily see who is involved with an API, and who they can reach out to with questions or feedback through the life of an API.

Github Repository

Github repositories can be used in conjunction with Postman throughout the API lifecycle, providing a way to keep artifacts synced, and automate the API lifecycle as part of the CI/CD process defined around a Github repository. Github repositories provides not just code source control, but also API source of truth as part of the deployment of APIs, that can be managed and kept in alignment with the rest of the API lifecycle using the Postman platform and seamless integrations with Github.

OpenAPI

The OpenAPI acts as the contract for any HTTP API, providing a machine readable way to describe the details of each request and response in a way that can be used across the entire API lifecycle. OpenAPI can be introduced in different ways via the API lifecycle, but once introduced should be managed as the source of truth for each API, always keeping up to date and managed via Postman workspace as well as synced to relevant repositories. The OpenAPI for each API can be used to generate collections for documentation, mocking, testing, and other areas of the API lifecycle, providing executable representations of each API tailored for a specific purpose, while maintaining a single source of truth for the contract of each API that can be used to keep docs, mocks, and tests in sync, but also to validate that each API contract is being met.

Test

A test-driven API lifecycle ensures that each API accomplishes the intended purpose it was developed for, providing manual and automated ways to ensure an API hasn't changed unexpectedly, is as performant as required, and meets the security expectations of everyone involved, helping establish a high quality of service consistently across all APIs.

Contract Testing

Contract testing ensures that each individual API and collection paths comply with the contract that was put forth for each individual version of an API. OpenAPI for each API can be used to generate collections providing 100% coverage for all of the API paths present. JSON Schema provides the details of the contract, and when applied as part of an OpenAPI definition, you end up with a machine readable contract that can be manually or automatically tested against. It is common for collections to be defined for the contract tests of each API, and then have these contracts tests scheduled for regular run via a monitor, as well as included as part of CI/CD pipelines, providing robust coverage for contract tests that developers can manually execute, or automated as required as part of the API lifecycle.

Monitor

All tests applied to an API should be monitored on a logical schedule and from relevant geographic regions, monitoring that APIs aren't breaking their contract, falling below their agreed upon service level agreement (SLA), or becoming a security risk, helping automate the quality of service across APIs in a way that allows teams to be as productive as possible.

Monitor

Postman monitors allow you to schedule the run of any collection, automating the run of API testing, governance, workflows, and anything else that is API-driven and can be defined as a collection. Monitors allow for the automation and orchestration of an almost infinite number of API capabilities, allowing for automating working with APIs, but also the API lifecycle. Monitors are how the API lifecycle becomes the moving API factory floor of an organization across different teams. API monitors run collections combined with different environments then allow the results to be published and piped into existing APM and other solutions, helping make API operations more observable.

Contract Testing Monitor

Monitors can be used to automate the scheduling of any contract test, taking any collection of API requests, complete with test scripts, combined with a targeted environment, and run in a variety of cloud regions on a recurring schedule. Providing the ability to independently execute the contract testing for each API, but then monitor the contract testing of each API over time to make sure nothing has changed. Providing the essential automation of contract testing needed to ensure it is consistently applied across all APIs, defining whether or not an API is not just meeting the uptime portion of its service level agreement, but actually is meeting business needs in an ongoing way.

Contract Testing Results

When each collection is executed by a monitor or pipeline it has a set of results that can be published to any location and saved for reporting upon and included in other dashboards. Having a strategy and standardized approach to handling the results of contract test runs helps inform individual developers and testers how they can handle test rules, and potentially allowing compliance with overall organizational API governance. Formalizing how test results inform API operations in real time, but also potentially providing historical data on the contract testing across teams and APIs.

Build

Tests can be automated using a CI/CD pipeline, allowing for test to be executed every time a commit or pull request is made against a repo. Allowing tests to be run against any instance of an API, ensuring that any API being deployed has not broken it's contract, and the API does what is expected.

CI/CD Pipeline

CI/CD pipelines are an essential part of automating the delivery of APIs, and there are multiple ways they can be used in conjunction with the Postman platform from deployment, to testing and governance. Postman collections are executable units that can be run individually and sequentially as part of a CI/CD pipeline, publishing documentation, running contract, integration, and performance tests, or leverage infrastructure APIs behind our APIs to automate any part of the API lifecycle. Jenkins, Github Actions, and other CI/CD solutions allow for the execution of manually created or dynamically generated Postman collections, providing an excellent mechanism for standardizing how API lifecycle automation occurs across many different APIs, while also utilizing a standardized approach to moving APIs forward as part of an agreed upon API lifecycle.

Newman

Newman is a command-line Collection Runner for Postman. It enables you to run and test a Postman Collection directly from the command line. It is built with extensibility in mind so that you can easily integrate it with your continuous integration servers and build systems. Newman maintains feature parity with Postman and allows you to run collections the way they are executed inside the collection runner in Postman. The open source Newman combined with the open source collection and environment provides a portable, executable unit of value that can be applied anywhere across API operations at the command line or pipeline layers. Providing for a robust way to apply API testing manually or in an automated way as part of the regular software development lifecycle, baking testing and governance into operations.

Development Environments

A development environment is a set of variables you can use in your Postman requests. You can use environments to group related sets of values together for use across individual, or many different collections. Environments help you abstract away secrets, tokens, and other data that you will need to use across different collections while working with different APIs. You can think of environments as a mini key-value store that you can apply to collections, but then also store data pulled from collection runs, persisting data across many different API requests. It is common for collection run results to be stored within environments, paginated data, and other results from API workflows and automation, making environments a pretty robust way to persist valuable data across API consumption.

Production Environments

A production environment is a set of variables you can use in your Postman requests. You can use production environments to group related sets of values together for use across individual, or many different collections. Environments help you abstract away secrets, tokens, and other data that you will need to use across different collections while working with different APIs. You can think of environments as a mini key-value store that you can apply to collections, but then also store data pulled from collection runs, persisting data across many different API requests. It is common for collection run results to be stored within environments, paginated data, and other results from API workflows and automation, making environments a pretty robust way to persist valuable data across API consumption.

Contract Testing

Contract testing ensures that each individual API and collection paths comply with the contract that was put forth for each individual version of an API. OpenAPI for each API can be used to generate collections providing 100% coverage for all of the API paths present. JSON Schema provides the details of the contract, and when applied as part of an OpenAPI definition, you end up with a machine readable contract that can be manually or automatically tested against. It is common for collections to be defined for the contract tests of each API, and then have these contracts tests scheduled for regular run via a monitor, as well as included as part of CI/CD pipelines, providing robust coverage for contract tests that developers can manually execute, or automated as required as part of the API lifecycle.

Contract Testing Results

When each collection is executed by a monitor or pipeline it has a set of results that can be published to any location and saved for reporting upon and included in other dashboards. Having a strategy and standardized approach to handling the results of contract test runs helps inform individual developers and testers how they can handle test rules, and potentially allowing compliance with overall organizational API governance. Formalizing how test results inform API operations in real time, but also potentially providing historical data on the contract testing across teams and APIs.

Observability

Tapping into the outputs available across API operations to understand what is happening with individual APIs throughout their lifecycle, but also in aggregate for domains, teams, and other logical groups, helping make API operations more visible in real time.

Activity

Tracking the activity that occurs across the design, development, and operation of APIs allows the ability to see what changes are made by teams to APIs and the supporting artifacts. Ensuring that there is observability present across the management of each API without teams having to do any additional work, tracking all changes, and then make this information visible to everyone involved. Providing an ongoing snapshot of the activity involved with moving each API forward, making sure nothing gets lost along the way. You can find an activity for each workspace using the overview tab and across workspaces via the home page of your Postman application.

Reports

Reports provide teams with visibility into the usage of individual APIs, but also the operations around them, helping us see activity, history, and the presence, or lack of presences of essential elements of the API lifecycle like documentation, testing, and other building blocks. The Postman platform provides native reporting on teams, APIs, and the activity that exists across workspaces. While also embracing an integration strategy that allows for the piping of data into other APM and observability solutions organizations are already using, providing another dimension for the creation of reports, and making them available via existing dashboards. Using API reporting to make APIs and API operations more visible in real time, allowing teams to stay aware of what is happening and be response to events that matter to an organization.

Contract Testing Monitor Report

With the run of each contract test monitor the results can be viewed using visualizer at run time, or the results can be saved, and reported upon later. Contract testing monitor results can be viewed internally within Postman and each contract testing collection using the visualizer tab and supporting test script, keeping reporting localized to each API and it’s tests. Contract testing monitor results can also be piped into other APM solutions using integrations, allowing for reporting to occur within dashboards within existing APM solutions used by teams.

Application Performance Management (APM)

Postman doesn’t reinvent the wheel when it comes to application performance management and embraces the use of existing APM solutions by making them a first-class integration within the platform, allowing you to integrate with Datadog, New Relic, and other APM solutions. Platform APM integrations will enable you to pipe the results of any collection run via a monitor into existing APM infrastructure. This approach to integration allows results of collection runs for contract, integration, performance, and other types of tests to be piped into APM solutions, providing visibility over the execution of any collection.

Conclusion

This blueprint is meant to provide a standardized approach to automating the testing of APIs across operations as part of a standardized API lifecycle. Each element within this blueprint works to provide a simple overview of what is involved across the entire life of an API, with more detail present on the detail page for each element (if you are viewing this on the API lifecycle project site). If you are reading this via a PDF or printed version you can visit the landing page for this blueprint to access more information and view specific actions you might possibly consider taking as part of applying each element of this proposed lifecycle within your own operations. This blueprint is a living document and will continue to evolve and be added to over time based upon feedback from readers. If you have any questions, feedback, or feel like there is more information you need, feel free to jump on the Github discussion for this blueprint, or any of the individual elements present--the value this blueprint provides is actively defined by the feedback community members like you.