Helios is now part of Snyk! Read the full announcement here.







How to adopt distributed tracing without compromising data privacy

Written by

Subscribe to our Blog

Get the Latest News and Content

Engineering teams can both drive productivity and comply with their company’s privacy policy when introducing distributed tracing into their tech stack to gain observability over microservices.


The age-old dilemma of privacy and security vs. productivity pops up for developers every time they consider introducing a new technology to their stack. The dilemma is often viewed as a trade-off: on one hand, privacy and security measures can slow down how quickly new features can be rolled out; on the other hand, prioritizing productivity and business enablement over privacy and security can increase the risk of breaches to an organization. The bottom line is that privacy and security live on a spectrum and every organization must decide for itself what its risk tolerance is.

Engineering decision-makers and CISOs have asked us about similar considerations when it comes to distributed tracing, especially for the purpose of microservices observability. When they consider introducing a new tool into their systems, they’re thinking of the overhead associated with the trust and accountability of the risks that tool can introduce. For example, installing an SDK in your application means you’re introducing code that is not your own into your system and it can potentially crash your app or simply have code vulnerabilities. This opens your application up to stability and security risks, not to mention possible risks associated with the exposure of sensitive customer data. Data collected for instrumentation may contain sensitive information such as user credentials, PII, or financial information. In light of these risks, engineers are asking questions about distributed tracing such as, “What is the risk that we’re introducing?” and “How much risk are we willing to tolerate?”

The answer doesn’t have to be all or nothing.

At Helios, we offer multiple implementation layers that enable organizations to drive dev velocity and engineering productivity, while complying with the organization’s risk profile. Helios is a dev-first observability platform that helps Dev and Ops teams shorten the time to find and fix issues in distributed applications. Built on OpenTelemetry, It provides traces and correlates them with logs and metrics, enabling end-to-end app visibility and faster troubleshooting. 

In this blog, we look specifically at the different layers Helios provides to help teams adopt distributed tracing and accelerate productivity while ensuring data privacy and compliance. The notion of layers is important as it allows the much-needed flexibility each team can leverage to comply with its own regulations. We deep dive into how each layer can be used to support different organizational risk profiles and business policies.


What are the privacy challenges of distributed tracing?

Instrumenting running application data to provide distributed tracing-driven observability, as is the case with any telemetry or monitoring system, could introduce risk around sensitive information being inadvertently captured and transmitted. Instrumentation involves the collection and transmission of large amounts of data across different components of a system, including sensitive data such as user IDs and authentication tokens. This data should be protected to prevent unauthorized access or breaches. Additionally, distributed tracing data may be subject to compliance regulations, such as GDPR or HIPAA, which require strict data protection and privacy measures.


So why would you want to implement distributed tracing?

Distributed tracing offers many benefits for improving the developer experience, especially when it comes to microservices observability. It provides valuable insights into system performance and behavior, helping developers reduce the mean time to resolution (MTTR) of issues in their distributed apps. Leveraging OpenTelemetry, tools like Helios provide granular visibility into traces, so you can understand the dependencies between different components in your application and how requests and data flow through your entire system. Leveraging such a widely adopted, industry-backed open source project such as OpenTelemetry reduced the risk of installing 3rd-party code by relying on the immense power and wisdom of the community to build the most robust, secure, and practical instrumentation solution. With Helios, you get end-to-end visibility into your application across microservices, serverless functions, databases, and 3rd party APIs, and you can see distributed tracing information in full context – including payloads, headers, and all span attributes. You can get insights as early as in your local and integration environment, all the way to production.


E2E distributed tracing leveraging OpenTelemetry with the full context in a single trace
An example for an E2E trace from the Helios Sandbox, providing the full context for the error that occurred in this deposit flow.


Adapting distributed tracing in line with your risk profile 

Helios’ mission is to help engineering teams adopt distributed tracing and OpenTelemetry so they benefit from enhanced developer productivity and best practices when it comes to microservices observability. Helios seeks to enable teams to achieve more with the resources they have, where they are in their growth journeys. To meet organizations where they are specifically in their risk profiles, Helios exposes various layers of control to set and select what data is instrumented in a system and how. The layers range from instrumenting all data, to collecting only metadata, and different options in between. 

The Helios SDK offers both allowlist and blocklist capabilities to ensure that the sensitive data collected complies with your company’s policies and needs. To comply with more strict security guidelines, the SDK also supports a mode in which only metadata is collected. Users can also choose to simply block specific paths from being collected altogether.

The table below outlines the layers Helios provides to engineering teams that wish to leverage distributed tracing while meeting privacy requirements:


Privacy policy Instrumentation layer Learn more
🟢  Flexible Instrument all data for optimal E2E visibility and complete troubleshooting and debugging coverage C++ | Erlang | Go | Java | .NET | Web JS | Node.js | Python | Ruby
⚪️  Diligent Obfuscate specific data, based on what’s configured specifically in the blocklist Data obfuscation blocklists
🟡  Moderate Obfuscate all data, except for what’s configured specifically in the allowlist Data obfuscation allowlists
🟠  Conservative Metadata-only mode ensures headers and payloads are not collected Metadata-only mode
🔴  Stringent Filter specific URLs and paths from being collected by the SDK altogether Data filtering


🟢  Flexible

For customers who have flexible privacy requirements, they can opt to have all their data instrumented for maximum E2E visibility and complete troubleshooting and debugging coverage. Customers can benefit from distributed tracing and observability over microservices in C++, Go, Java, .NET, Python, and other languages. They can drill down to services, view specific requests, and gain additional context with enriched OpenTelemetry instrumentation. Helios delivers all the context needed for troubleshooting of issues, including all payloads (e.g. HTTP request/response bodies, message queues content, DB queries and results, etc.) headers, and all span attributes.

Response body of all recent API calls in the trace visualization
An example from the Helios Sandbox for how easily accessible the response body of recent API calls is available for fast troubleshooting, when leveraging the full instrumentation Helios offers.


⚪️  Diligent

Customers who are asked to apply privacy practices more diligently can obfuscate specific data according to a configured blocklist. They can set what specific data must be obfuscated by the Helios SDK, and only data that matches an expression in the JSONPaths configured will be collected in a secret manner. For example, a user can mark sensitive information like first name, last name, and credit card number to be obfuscated. 


🟡  Moderate

If customers go one level more cautious with their privacy requirements, they can choose to obfuscate all data by default, except for what’s configured in an allowlist. Customers can put into an allowlist specific data that does not need to be obfuscated by the Helios SDK. Only data that matches an expression in the JSONPaths configured will be collected and sent as is.


🟠  Conservative

Customers with more conservative privacy requirements can can choose to receive metadata only, which means that the Helios SDK will take all the payload data and neutralize it, leaving only the metadata that OpenTelemetry provides. When the metadata-only mode is configured, no content of headers or payloads (including HTTP request/response bodies, message queues content, DB queries and results, etc.) is collected. 

Recent API calls based on basic OpenTelemetry instrumentation without payloads
An example from the Helios OpenTelemetry Sandbox for metadata-only instrumentation – all recent API calls are shown and the E2E traces are available, but less context exists for fast troublehsooting.


🔴  Stringent

In the face of the most extreme privacy requirements, customers can choose to drop certain traffic & data altogether. They can filter specific URLs and paths from being collected by the Helios SDK. Users can control what data is being traced and reported to Helios; if a user marks something as sensitive, no part of it will be collected by the SDK.



Distributed tracing improves engineering productivity and developer experience by accelerating dev velocity and driving engineering excellence, especially when it comes to microservices observability. With distributed tracing, engineering teams get the observability insights they need, when they need them most, to troubleshoot and improve the performance of their apps. To benefit from distributed tracing, teams don’t need to forego privacy requirements. Balancing between developer productivity and security is ultimately a tradeoff, and Helios provides layers you can use to ensure you’re taking advantage of distributed tracing insights without sacrificing the business’ data privacy policies. Naturally, instrumenting all data and providing the complete context lead to faster results and ultimately reduce MTTR. Helios allows developers to be in control of what data is instrumented in their applications and how. This is one more way in which we wish to see more and more teams embrace distributed tracing and do more with the resources they have.

Subscribe to our Blog

Get the Latest News and Content

About Helios

Helios is an applied observability platform that produces actionable security and monitoring insights. We apply our deep runtime data collection capabilities to help Sec, Dev, and Ops teams understand the actual application risk posture, prioritize vulnerabilities, shorten troubleshooting time, and reduce MTTR.

The Author

Maya Mandel
Maya Mandel

Maya is the VP Product at Helios. She has vast experience in product management and leadership, including building the product teams at Skycure (acquired by Symantec) and Viz.ai. Maya is a former Captain in the IDF intelligence unit 8153, holds a BSE in Computer Science from the University of Michigan (where she was also a member of the Women's Swimming & Diving team) and an MBA from Columbia Business School.

Related Content

Challenges of existing SCA tools
Challenges with Traditional SCA Tools
Application security testing tools are designed to ensure that applications are put through rigorous security assessments to identify security flaws within...
Read More
Banner for blog post - Scaling microservices - Challenges, best practices and tools
The Challenges of Collecting Runtime Data
Collecting data in real-time plays a crucial role in securing, monitoring, and troubleshooting applications. This real-time data, often referred to as...
Read More
Helios Runtime for Appsec
Helios Runtime for AppSec: The missing link in application security
Modern development teams increasingly rely on open-source packages to rapidly build and deploy applications. In fact, most, if not all applications consist...
Read More