Blueprint: Test Automation

Test automation is essential for delivering reliable and consistent APIs at scale across the enterprise. This is a blueprint for walking through the base of how test automation can work as part of a well defined API lifecycle, helping teams standardize how they approach testing APIs.

This blueprint focuses just on the test automation portion of the API lifecycle, highlighting the different types of tests that can be run, how monitors can be used to automate those tests on a schedule from multiple regions, and then how to tap into the observability opportunities that exist when it comes to test automation. Test automation should provide you with a 50K foot view of how testing works in conjunction with monitors, and what observability is available by default, published using visualizer, or introduced through integrations with external solutions, providing many different ways of being able to see and understand test automation for API operations.


Define

The most important first step of any API lifecycle is to make sure the operations around an API are properly defined, laying the foundation for being able to effectively design and bring an API to life, while also establishing a known place, or places to go to get all the information you need regarding each individual API, or groups of APIs. A little planning and organization at this early step of the API journey can go a long way towards ensuring the overall health and velocity of an API, and the applications and integrations that will depend on each internal, partner, or public API being delivered.

  • Team Workspace - Establishing and properly setting up a dedicated workspace for each API helps ensure there is always a single place to go to find everything that is happening with an API across it's entire lifecycle.
  • Team Members - Formerly defining who the team will be moving an API forward through all stages of it's lifecycle, providing a clear definition of who is responsible for each part of producing an API.
  • Github Repository - Having a dedicated Github repository for an API provides a single place where code and other artifacts can be managed, with a pipeline, issues, and other supporting elements an API will need to operate.
  • OpenAPI - The OpenAPI specification provides a common vocabulary for describing the surface area of request and response APIs, as well as webhooks, ensuring that API producers and consumers are on the same page when it comes to integration, but also helps stabilize multiple areas of the API lifecycle providing a contract that can be used for delivering documentation, mocks, testing, and more.

Test

A test-driven API lifecycle ensures that each API accomplishes the intended purpose it was developed for, providing manual and automated ways to ensure an API hasn't changed unexpectedly, is as performant as required, and meets the security expectations of everyone involved, helping establish a high quality of service consistently across all APIs.

  • Contract Testing - Contract tests can be derived from API artifacts like OpenAPI, JSON Schema, and GraphQL, and used to ensure there are no changes to what has been promised when an API was designed or developed, providing a repeatable test that can be run manually, on a schedule from multiple regions, or as part of a CI/CD pipeline to ensure contracts with consumers are upheld.

Monitor

All tests applied to an API should be monitored on a logical schedule and from relevant geographic regions, monitoring that APIs aren't breaking their contract, falling below their agreed upon service level agreement (SLA), or becoming a security risk, helping automate the quality of service across APIs in a way that allows teams to be as productive as possible.

  • Monitor - Monitoring any process across API operations defined as a collection, then bundled with any environment, setting a schedule for the execution of the collection, and viewing or publishing of the results to any other location, providing a very wide definition of what monitoring can mean across API operations.
  • Contract Testing Monitor - Monitors can be setup for all API contract testing, scheduling the run of contract testing when needed and configuring to run from the regions that matter most to business operations, making sure that each individual API contract is never breaking with consumers, making sure that each API is delivering as expected on a 24/7 schedule.
  • Contract Testing Results - The results of any contract testing run can be rendered as test results, presented using visualizer, and posted to existing APM or other destination that can be used as part of wider observability systems, allowing contract tests across all APIs to be centralized for understanding API performance at scale.

Build

Tests can be automated using a CI/CD pipeline, allowing for test to be executed every time a commit or pull request is made against a repo. Allowing tests to be run against any instance of an API, ensuring that any API being deployed has not broken it's contract, and the API does what is expected.

  • CI/CD Pipeline - Like other aspects of the software development lifecycle, APIs benefit from well defined, repeatable CI/CD pipelines to deploy, test, automate, and orchestrate throughout the API lifecycle, publishing APIs to development, staging, production, and other stages, but then also ensuring other elements like publishing of documentation, and tests are baked into the process.
  • Newman - Newman is an open source collection runner, allowing Postman collections to be executed as part of CI/CD pipeline runs, executing all of the API requests, pre-request, and post-request scripts, and providing results reporting that can be used as part of testing and automation.
  • Development Environments - Machine readable environments for APIs allow for abstracting away common elements of a API development environment from the definition of each API, allowing different environments to be paired with collections for each API at design, develop, and build time.
  • Production Environments - Machine readable environments for APIs allow for abstracting away common elements of an API production environments from the definition of each API, allowing different environments to be paired with collections for each API at design, develop, and build time.
  • Contract Testing - Contract tests can be derived from API artifacts like OpenAPI, JSON Schema, and GraphQL, and used to ensure there are no changes to what has been promised when an API was designed or developed, providing a repeatable test that can be run manually, on a schedule from multiple regions, or as part of a CI/CD pipeline to ensure contracts with consumers are upheld.
  • Contract Testing Results - The results of any contract testing run can be rendered as test results, presented using visualizer, and posted to existing APM or other destination that can be used as part of wider observability systems, allowing contract tests across all APIs to be centralized for understanding API performance at scale.

Observability

Tapping into the outputs available across API operations to understand what is happening with individual APIs throughout their lifecycle, but also in aggregate for domains, teams, and other logical groups, helping make API operations more visible in real time.

  • Activity - The changes made to any aspect of operations by team members, providing observability into when APIs, mock servers, documentation, testing, monitors, and other critical elements of API operations are changed, configured-- helping give a log of everything that happens at the operational level.
  • Reports - Visual reports that aggregate data from across operations, making APIs and the operations around them something that team members can see activity, hisotry, and other dimension of what is happening across API operations, allowing different views to be organized and presented via dashboards.
  • Contract Testing Monitor Report - A report that allows the one time or historical results of contract testing monitors to br visualized, providing visibility into whether or not an API contract has been broken at any point in time as defined the schedule of the contract testing monitor runs.
  • Application Performance Management (APM) - The monitoring and management of performance and availability of APIs, actively detecting and diagnosing performance problems to maintain an expected level of service, translating API operational metrics into business meaning value at scale across hundreds or thousands of internal or external APIs.

This blueprint is meant to provide a standardized approach to automating the testing of APIs across operations as part of a standardized API lifecycle. Each element within this blueprint works to provide a simple overview of what is involved across the entire life of an API, with more detail present on the detail page for each element (if you are viewing this on the API lifecycle project site). If you are reading this via a PDF or printed version you can visit the landing page for this blueprint to access more information and view specific actions you might possibly consider taking as part of applying each element of this proposed lifecycle within your own operations. This blueprint is a living document and will continue to evolve and be added to over time based upon feedback from readers. If you have any questions, feedback, or feel like there is more information you need, feel free to jump on the Github discussion for this blueprint, or any of the individual elements present--the value this blueprint provides is actively defined by the feedback community members like you.


Discussions

YAML

Printable

Return to Main Page

This provides a link back to the home page if you need it.