Helios is now part of Snyk! Read the full announcement here.







Testing Microservices – Trace Based Integration Testing Example

Written by

Subscribe to our Blog

Get the Latest News and Content

As engineering organizations transitioned from monolith to microservices architectures, they sought to make their development efforts more scalable and manageable. The microservices paradigm promised to increase development speed, reduce MTTR and improve quality while cutting down on maintenance costs. However, in real life, there are inherent quality caveats when it comes to developing microservices.

Challenges in Testing Microservices – Integration Chaos

One of the main challenges developers face is with microservices testing. Microservices are much more difficult and complex to test than monolith architectures. This is especially true when attempting to replicate traditional testing methods to microservices. Unlike a monolith, the microservices architecture is made up of separate services that communicate with each other. While they function in production as related units, they are developed as independent units in different teams.

So while service A depends on services B – E when in production, this is often not reflected in the testing environment. If a developer runs traditional test methods for component A, the testing environment will not always include the up-to-date versions of services B – E, but perhaps older or incomplete versions of them (i.e., a mock). This means service A isn’t being accurately tested for real-life, production behavior, and changes that are made to it might cause unexpected downstream behavior.

In order to minimize bugs in production and increase developer velocity and confidence, microservices testing has to evolve and become more reliable.

Solving Microservices Testing Pain with OpenTelemtry

What is openTelemetry? OpenTelemetry (also called OTel) is an observability open-source framework for microservices and distributed apps- that enables generating and capturing telemetry data from cloud-native microservices software. OpenTelemetry supports the full range of observability signals across traces, metrics, and logs. 

For the purpose of visualization of the flow through a system, OTel distributed tracing is extremely relevant. It is being used by tier 1 companies such as Google, Shopify, AWS, Datadog, Microsoft, and others. OTEL tracing shows the exact steps a transaction takes through with needed data, such as the calls, response codes, and timing. 

However, OpenTelemetry on its own doesn’t provide advanced capabilities on top of the raw data and is therefore enhanced by modern dev tools.  Modern tools for microservices trace-based testing are utilizing OpenTelemetry to prevent issues, by using it in the integration test environment.

These tools are enriching OpenTelemetry with observability, debugging, and testing (2E and integration testing). In other words, OpenTelemetry traces are used to create integration tests with more details and precision than ever before. This approach is called trace-based testing. 

I Co-Founded Helios to solve a significant pain utilizing OpenTelemetry. It is a developer platform designed specifically for testing and troubleshooting microservices. By leveraging distributed tracing, Helios provides granular visibility into the entire microservices architecture and into each service. Such capabilities aren’t available in OpenTelemetry as is. 

This data is then used to automate the creation of microservices tests and to provide insights into specific failures and bottlenecks. It allows developers to go back to relying on testing as an essential and trustworthy step for delivering production-ready code.


Microservices Integration-Testing – A Step-by-Step Walkthrough

Let’s see how it works.

Take a look at a typical testing scenario. Say we’re building a financial system that allows people to transfer money to each other, like PayPal or Venmo. A typical architecture includes the following components:

  • API service – An HTTP server that behaves as the system gateway, exposing the system’s external APIs
  • Users service – An HTTP server that manages all users and their permissions
  • Accounts service – A Kafka consumer that manages all account transactions
  • Payments service – An HTTP server working with 3rd party payments APIs (e.g., Stripe)
  • ֿFinancial service – An HTTP server that handles exchange rates by fetching them from 3rd party financial APIs
  • Emails service – A Kafka consumer that sends emails on different events in the system

In our scenario, the API service exposes a “/deposit” endpoint, which receives the receiving user’s email, currency, and amount. Say we’re tasked with adding a test that verifies a new capability – whenever a successful API call to “/deposit” is made, the email service should send an email to the receiving user, stating that a transaction to his/her accounts has been made successfully.

Why Traditional Testing Methods Fail when Microservices are Involved? 

While this is a typical scenario, it’s not that easy to test with traditional testing methods that were designed for different needs. Challenges include:

  • Validating the email service behavior – The email service is a Kafka consumer, which means there’s no synchronous indication of its success/failure. How can we tell if the test passed and if the behavior we’re trying to validate actually occurred?
  • Generating the correct API payload – With hundreds (if not thousands) of API endpoints, generating the right payload for the API we’d like to test is like finding a needle in a haystack.
  • A “/deposit” request cannot be made to a non-existent user – The scenario requires a setup, i.e an API request that creates the receiving user. This requires more tinkering in the architecture before we can run the test.
  • Identifier creation – The test requests must share the same identifiers: the email, user id, etc. These need to be added. Moreover, the user id is generated dynamically as part of the test, and therefore cannot be hard-coded. How can we ensure all identifiers are correctly added in the right places?

The Right Way- Automated Test Generation and Customization in Microservices

We, therefore, designed a solution that automates the creation of this testing scenario while skipping over the hurdles a developer would have to figure out manually when testing with legacy solutions:

Step 1: Create a “setup action” for creating a user

First, navigate the system’s API to find the user creation request

testing microservices with Helios

Click on “Add as setup action”, and customize it. Helios supports parameterizing any parameters of the HTTP request by using the {{$var_name}} syntax. In the image below, you can see that two variables are defined: hostname and userEmail. Their value is defined globally and can be easily replaced.

testing microservices with Helios

Step 2: Create your test

Create an E2E test from the “/deposit” endpoint:

testing microservices with Helios

Afterward, add the “Create user” setup action to the “/deposit” E2E test by adding it in the left pane menu:

testing microservices with Helios

You can see that a “setup action” node was added to the graph. After clicking on it, the right pane menu contains the setup action config that we just created.

Step 3: Customize

Let’s click on the “/deposit” request edge and customize it – we need to apply the hostname and userEmail variables, to make sure they are correlated with the setup action:

Testing microservices with Helios

Step 4: Generate test code

We can now generate the test code and see that all the variables are correlated:

testing microservices with Helios

Step 5: Validate email service behavior

Remember that we wanted to validate that an email was indeed sent from the emails service? We simply need to select the SES operation in the graph and configure the validation accordingly. Notice that we’re using the userEmail variable here as well.

Testing microservices with Helios

During the test, Helios also collects all the payloads automatically and regenerates the same API calls, including parameterization.

Advanced Testing Capabilities – Dynamic Variables

For more advanced test customization needs, the tool also provides the ability to add dynamic variables, i.e., variables whose values are populated as part of the test execution.

It supports extracting parameters from a setup action and using it at a later stage of the test. A typical example would be using the user ID that returns as a response from the “Create user” setup action and using it to make a query on the user’s balance.

To use dynamic variables, simply go the response of the setup action, select the “custom” validation and extract the variable:

Testing microservices with Helios

Then, you can use this variable anywhere you like; in the case below, as part of the URL that queries the user’s balance:

Testing microservices with Helios






















Learn more about setting up actions when building complex E2E tests. 

Ready to get started with your own testing scenario? Automate your microservices tests for free. 


Python Instrumentation and testing

Go Instrumentation


This article addresses the differences between traditional test-driven development vs the new observability-driven development, While the first fails when it comes to microservices, the latter offers a good solution. Microservices testing best practices rely on modern dev tools. These are Visual, trace-based integration testing tools, that are based on open-source such as OpenTelemetry, allowing developers to automate integration testing easily, as seen in the example above. 

Subscribe to our Blog

Get the Latest News and Content

About Helios

Helios is an applied observability platform that produces actionable security and monitoring insights. We apply our deep runtime data collection capabilities to help Sec, Dev, and Ops teams understand the actual application risk posture, prioritize vulnerabilities, shorten troubleshooting time, and reduce MTTR.

The Author

Ran Nozik
Ran Nozik

CTO and co-founder of Helios. An experienced R&D leader, and mathematician at heart. Passionate about improving modern software development, and a big fan of and contributor to OpenTelemetry. After serving as an officer in unit 8200 and leading R&D efforts in the cybersecurity department, working as a Senior Software Developer, and becoming an Engineering Team Leader, Ran co-founded Helios, a production-readiness platform for developers. Ran holds a B.Sc. in Computer Science and Mathematics from the Hebrew University of Jerusalem.

Related Content

Challenges of existing SCA tools
Challenges with Traditional SCA Tools
Application security testing tools are designed to ensure that applications are put through rigorous security assessments to identify security flaws within...
Read More
Banner for blog post - Scaling microservices - Challenges, best practices and tools
The Challenges of Collecting Runtime Data
Collecting data in real-time plays a crucial role in securing, monitoring, and troubleshooting applications. This real-time data, often referred to as...
Read More
Helios Runtime for Appsec
Helios Runtime for AppSec: The missing link in application security
Modern development teams increasingly rely on open-source packages to rapidly build and deploy applications. In fact, most, if not all applications consist...
Read More