As engineering organizations transitioned from monolith to microservices architectures, they sought to make their development efforts more scalable and manageable. The microservices paradigm promised to increase development speed, reduce MTTR and improve quality while cutting down on maintenance costs. However, in real life, there are inherent quality caveats when it comes to developing microservices.
Challenges in Testing Microservices – Integration Chaos
One of the main challenges developers face is with microservices testing. Microservices are much more difficult and complex to test than monolith architectures. This is especially true when attempting to replicate traditional testing methods to microservices. Unlike a monolith, the microservices architecture is made up of separate services that communicate with each other. While they function in production as related units, they are developed as independent units in different teams.
So while service A depends on services B – E when in production, this is often not reflected in the testing environment. If a developer runs traditional test methods for component A, the testing environment will not always include the up-to-date versions of services B – E, but perhaps older or incomplete versions of them (i.e., a mock). This means service A isn’t being accurately tested for real-life, production behavior, and changes that are made to it might cause unexpected downstream behavior.
In order to minimize bugs in production and increase developer velocity and confidence, microservices testing has to evolve and become more reliable.
Solving Microservices Testing Pain with OpenTelemtry
What is openTelemetry? OpenTelemetry (also called OTel) is an observability open-source framework for microservices and distributed apps- that enables generating and capturing telemetry data from cloud-native microservices software. OpenTelemetry supports the full range of observability signals across traces, metrics, and logs.
For the purpose of visualization of the flow through a system, OTel distributed tracing is extremely relevant. It is being used by tier 1 companies such as Google, Shopify, AWS, Datadog, Microsoft, and others. OTEL tracing shows the exact steps a transaction takes through with needed data, such as the calls, response codes, and timing.
However, OpenTelemetry on its own doesn’t provide advanced capabilities on top of the raw data and is therefore enhanced by modern dev tools. Modern tools for microservices trace-based testing are utilizing OpenTelemetry to prevent issues, by using it in the integration test environment.
These tools are enriching OpenTelemetry with observability, debugging, and testing (2E and integration testing). In other words, OpenTelemetry traces are used to create integration tests with more details and precision than ever before. This approach is called trace-based testing.
I Co-Founded Helios to solve a significant pain utilizing OpenTelemetry. It is a developer platform designed specifically for testing and troubleshooting microservices. By leveraging distributed tracing, Helios provides granular visibility into the entire microservices architecture and into each service. Such capabilities aren’t available in OpenTelemetry as is.
This data is then used to automate the creation of microservices tests and to provide insights into specific failures and bottlenecks. It allows developers to go back to relying on testing as an essential and trustworthy step for delivering production-ready code.
Microservices Integration-Testing – A Step-by-Step Walkthrough
Let’s see how it works.
Take a look at a typical testing scenario. Say we’re building a financial system that allows people to transfer money to each other, like PayPal or Venmo. A typical architecture includes the following components:
- API service – An HTTP server that behaves as the system gateway, exposing the system’s external APIs
- Users service – An HTTP server that manages all users and their permissions
- Accounts service – A Kafka consumer that manages all account transactions
- Payments service – An HTTP server working with 3rd party payments APIs (e.g., Stripe)
- ֿFinancial service – An HTTP server that handles exchange rates by fetching them from 3rd party financial APIs
- Emails service – A Kafka consumer that sends emails on different events in the system
In our scenario, the API service exposes a “/deposit” endpoint, which receives the receiving user’s email, currency, and amount. Say we’re tasked with adding a test that verifies a new capability – whenever a successful API call to “/deposit” is made, the email service should send an email to the receiving user, stating that a transaction to his/her accounts has been made successfully.
Why Traditional Testing Methods Fail when Microservices are Involved?
While this is a typical scenario, it’s not that easy to test with traditional testing methods that were designed for different needs. Challenges include:
- Validating the email service behavior – The email service is a Kafka consumer, which means there’s no synchronous indication of its success/failure. How can we tell if the test passed and if the behavior we’re trying to validate actually occurred?
- Generating the correct API payload – With hundreds (if not thousands) of API endpoints, generating the right payload for the API we’d like to test is like finding a needle in a haystack.
- A “/deposit” request cannot be made to a non-existent user – The scenario requires a setup, i.e an API request that creates the receiving user. This requires more tinkering in the architecture before we can run the test.
- Identifier creation – The test requests must share the same identifiers: the email, user id, etc. These need to be added. Moreover, the user id is generated dynamically as part of the test, and therefore cannot be hard-coded. How can we ensure all identifiers are correctly added in the right places?
The Right Way- Automated Test Generation and Customization in Microservices
We, therefore, designed a solution that automates the creation of this testing scenario while skipping over the hurdles a developer would have to figure out manually when testing with legacy solutions:
Step 1: Create a “setup action” for creating a user
First, navigate the system’s API to find the user creation request
Click on “Add as setup action”, and customize it. Helios supports parameterizing any parameters of the HTTP request by using the {{$var_name}} syntax. In the image below, you can see that two variables are defined: hostname and userEmail. Their value is defined globally and can be easily replaced.
Step 2: Create your test
Create an E2E test from the “/deposit” endpoint:
Afterward, add the “Create user” setup action to the “/deposit” E2E test by adding it in the left pane menu:
You can see that a “setup action” node was added to the graph. After clicking on it, the right pane menu contains the setup action config that we just created.
Step 3: Customize
Let’s click on the “/deposit” request edge and customize it – we need to apply the hostname and userEmail variables, to make sure they are correlated with the setup action:
Step 4: Generate test code
We can now generate the test code and see that all the variables are correlated:
Step 5: Validate email service behavior
Remember that we wanted to validate that an email was indeed sent from the emails service? We simply need to select the SES operation in the graph and configure the validation accordingly. Notice that we’re using the userEmail variable here as well.
During the test, Helios also collects all the payloads automatically and regenerates the same API calls, including parameterization.
Advanced Testing Capabilities – Dynamic Variables
For more advanced test customization needs, the tool also provides the ability to add dynamic variables, i.e., variables whose values are populated as part of the test execution.
It supports extracting parameters from a setup action and using it at a later stage of the test. A typical example would be using the user ID that returns as a response from the “Create user” setup action and using it to make a query on the user’s balance.
To use dynamic variables, simply go the response of the setup action, select the “custom” validation and extract the variable:
Then, you can use this variable anywhere you like; in the case below, as part of the URL that queries the user’s balance:
Learn more about setting up actions when building complex E2E tests.
Ready to get started with your own testing scenario? Automate your microservices tests for free.
Related:
Python Instrumentation and testing
Conclusion
This article addresses the differences between traditional test-driven development vs the new observability-driven development, While the first fails when it comes to microservices, the latter offers a good solution. Microservices testing best practices rely on modern dev tools. These are Visual, trace-based integration testing tools, that are based on open-source such as OpenTelemetry, allowing developers to automate integration testing easily, as seen in the example above.
