Helios is now part of Snyk! Read the full announcement here.







Trace-based Testing: Modern Testing for the Modern Infrastructure

Written by

Subscribe to our Blog

Get the Latest News and Content

Traditional software application infrastructure generally followed a monolithic pattern; one application connected to one database. Understanding what was going on inside the application itself was a simple proposition, with relevant logs and metrics all originating from the same source. When it came time to scale, the answer was to get a bigger server or to get more servers.

In contrast, modern software application infrastructure distributes the logical parts of an application across workloads, platforms, and even data centers and cloud providers. Therefore, traditional, monolithic-focused monitoring tools cannot provide a complete picture of what’s going on inside an application. Engineering teams that don’t have a full understanding of their architecture will be slow to iterate and release new features, putting business outcomes in jeopardy. So the question becomes: what can provide that complete picture?  

When most engineers familiar with web-facing software think of monitoring, they think of metrics, logging, and traditional performance metrics like response time. However, these measures don’t tell the whole story of distributed app performance and behavior. This is where tracing can help.

What is Tracing?

Tracing is generally defined as the act of following a user request through a multi-service architecture. Tracing is a necessity to gain a holistic understanding of application behavior, particularly applications on distributed infrastructure.

Tracing can further be broken down into Spans. The first Span of a trace is called the Root Span, which triggers the entire request/trace. Spans triggered by the Root Span represent more granular parts of the request. Spans are typically associated with individual URIs or services that take part in the larger request context, such as authentication.

What is Trace-based Testing?

Tracing/ distributed tracing is a great visibility tool for understanding application behavior. However, it comes with a caveat; it’s difficult for developers to take advantage of the benefits offered by tracing while working in a distributed system, where traditional testing methodologies may not apply.

Fortunately, trace-based testing fills that gap. Trace-based testing is a form of software testing that utilizes the core functionality of distributed tracing in order to produce more comprehensive, application-wide test results and data. 

Trace-based testing allows developers to make validations based on tracing data. Furthermore, trace-based testing can be integrated with existing testing frameworks to provide specific data and query validations throughout the lifecycle of a request. This enhanced visibility becomes possible via test logic that extends beyond the boundaries of an individual function or application component.

As a result, trace-based testing also improves the overall functionality and usefulness of application testing. Consider unit tests: it’s often left to developers to enumerate various aspects of how their application is expected to behave in a given scenario, and then to create tests to validate that behavior. Unit testing helps improve code quality, but testing complex application logic is difficult and incurs a cognitive tax on the developer. With trace-based testing, developers can still be productive with unit-testing practices they know, but with the enhanced capabilities offered by tracing features added on.

Finally, trace-based testing provides fast feedback, for improved development patterns. Immediate and actionable feedback from trace-based testing empowers developers to quickly gain an understanding of how their proposed changes impact application behavior and allows them to iterate on and deliver features faster.

Trace-based Testing in Action

A great way to highlight the benefits of trace-based testing is to see it in action. In this example, the application is a basic web service for providing restaurant recommendations. The application uses Flask as a frontend request router, API, and web server. Backend data storage is provided by MongoDB, which has been pre-populated with example data. Frontend requests are also routed to a Kafka server for later review or replay. For development, everything is running in local Docker containers.

Traditional Testing

Before getting into setting up trace-based testing, it will be helpful to understand how developers might use traditional testing methods in this scenario. Consider one of the application’s API endpoints:

trace based testing, example

Typically, a developer will need to spend time manually designing and implementing tests that will cover as much application behavior as possible. For this use case, Flask applications can take advantage of the pytest framework with fixtures. A developer can use fixtures to set up a mock client, run tests, and then tear down any resources that were used after the tests have concluded.

For the example application, a developer might create this test for the /fivestar endpoint:

def test_get_restaurant(client):
response = client.get("/fivestar")
assert response.data['rating'] == 5

This test checks that the response from the application indeed returns a restaurant with a rating of 5. However, because this is a fixture, this is not occurring in the context of a live application. The developer will also need to create mock data fixtures to mimic the behavior of the backend database.

The problems with the traditional testing approach are becoming plainly obvious: developers have to spend critical cycles manually crafting tests that only emulate application behavior in a sterile environment. Even with these tests, the developer still does not have a clear picture of how a customer request will actually behave in the application infrastructure.

Trace-based Testing: A Tutorial

With trace-based testing, developers have a more comprehensive view of their applications. Let’s see how it works with Helios.

Using the button in the upper left, we can add services to our environment:

trace based testing, example

Since the example application uses Flask, a Python module, Python is the obvious choice for the service:

trace based testing, example

The environment variables need to be available during application runtime. For Docker-based applications, these are commonly placed in the Dockerfile. Once the variables have been populated, and the application started, there still won’t be any tracing data visible. The missing element is some actual requests.

The demo application will have three distinct services: frontend, API, and web. All requests will pass through the frontend to either the API or web endpoint. All requests will also be recorded in a Kafka queue called “review-app”.

Requests to the API for a randomized 5-star restaurant:

$ curl

Request to the API for all restaurants serving pizza:

$ curl

And then make a visit to the 5-star and cuisine endpoints in the browser:

trace based testing, example

trace based testing, example

Now there should be several services present in the UI, along with their recent request traces:

trace based testing, example

Individual requests, shown as “operations”, show a data-rich, user-friendly representation of individual requests as they traverse spans and nodes:

With this data available, developers can get real-time insights into application behavior and performance. Clicking through one of the traces presents an extensive, in-depth picture of the entire lifecycle of the request:

trace based testing, example

Even though this is being handled via UI, Helios makes it possible to generate repeatable test automation, which gives development teams the capability to standardize and scale trace-based testing for the entire stack. 

Switching to the “Test” interface allows users to create programmatic test fixtures with options for defining query parameters and expected return values:

trace based testing, example

Clicking “Generate test code” generates fully functional tests for common language testing modules, in this case, pytest:

Developers can copy this code to existing test libraries, which can be reused across different deployments. To further extend testing capabilities, Helios also lets developers perform validations during tracing tests. Looking at the “Test” tab for one of the five-star endpoint traces:

trace based testing, example 

The empty blue circle on the span leading to the MongoDB node is a validation checkpoint. Enabling this checkpoint provides a regex validation mechanism for query and return values:

trace based testing, example

A good validation for the `/fivestar` endpoint would be to assert that the return value has the expected value of “5” for the “rating” field. The Data validation should be configured as follows:

trace based testing, example

Clicking “Generate test code” produces a complete test, including the validation assertion:trace based testing, example

This code needs to be copied to the local development test suite. For the purposes of this demonstration, the test can be placed in the root directory in a file called `test_api.py`. The Helios docs provide instructions for running this test with the pytest module. A successful test run will produce a URL linking to the results in the Helios UI. Clicking that URL shows the specific test invocation:

trace based testing, example

Clicking on the test shows a new trace result, with the test results as added context:

trace based testing, example 

The ability to seamlessly interweave tracing and programmatic tests provides detailed, data-driven insights into application behavior that just isn’t possible with traditional testing methods. 

Trace-based Testing Wrap-up

Modern application infrastructure is complex. Traditional testing, such as unit tests, are good development practices, but they require extensive work by development teams to provide even partial testing coverage for a complex, distributed application.

Trace-based testing helps developers understand this complexity, and gives developers confidence in their test suites, and ultimately their application code. Automated trace-based testing solutions integrate with existing developer workflows in a frictionless way; avoiding the need to rely on ops teams or heavy modifications to application dependencies. By providing visibility into application behavior and architecture components, trace-based testing provides immediate, actionable feedback for developers to improve their applications and critically, to improve the overall customer experience.

To get started with trace-based testing for free with Helios, click here.

Subscribe to our Blog

Get the Latest News and Content

About Helios

Helios is an applied observability platform that produces actionable security and monitoring insights. We apply our deep runtime data collection capabilities to help Sec, Dev, and Ops teams understand the actual application risk posture, prioritize vulnerabilities, shorten troubleshooting time, and reduce MTTR.

The Author


Helios is an applied observability platform that produces actionable security and monitoring insights. We apply our deep runtime data collection capabilities to help Sec, Dev, and Ops teams understand the actual application risk posture, prioritize vulnerabilities, shorten troubleshooting time, and reduce MTTR.

Related Content

Challenges of existing SCA tools
Challenges with Traditional SCA Tools
Application security testing tools are designed to ensure that applications are put through rigorous security assessments to identify security flaws within...
Read More
Banner for blog post - Scaling microservices - Challenges, best practices and tools
The Challenges of Collecting Runtime Data
Collecting data in real-time plays a crucial role in securing, monitoring, and troubleshooting applications. This real-time data, often referred to as...
Read More
Helios Runtime for Appsec
Helios Runtime for AppSec: The missing link in application security
Modern development teams increasingly rely on open-source packages to rapidly build and deploy applications. In fact, most, if not all applications consist...
Read More