Modern applications are becoming increasingly distributed due to a wide range of benefits including enhanced scalability, high availability, fault tolerance, and better geographical distribution. But it also makes the overall system complex making it challenging to understand how they function internally. Distributed tracing helps to address it by tracking how requests flow through various system components with detailed insights.
Over time, several tools and platforms have arisen to assist with implementing distributed tracing. However, implementing observability in distributed systems has proved to be a significant challenge. This is because there is a lack of understanding between developers who create libraries and frameworks and vendors who design observability tools for distributed tracing. As a result, wrappers and adapters are required to make them interoperable, resulting in numerous inefficiencies.
To address this issue, OpenTelemetry (OTel) emerged as an industry standard to facilitate distributed tracing.
OTel in a Nutshell
Before OTel, there were many standards in the observability space by open source and third-party vendors. Out of them OpenCensus and OpenTracing have emerged from the open-source community focusing on solving a specific part of the problem. Later, these two projects were combined to create a unified standard, which gave rise to OTel.
OTel comes with a novel approach, where a library must produce all the Traces and Metrics OTel provides. This approach comes with several benefits that make it:
- Straightforward for library and framework developers to use.
- It makes it vendor-neutral.
- Provides consistent performance for the implementation.
OTel already supports all the major programming languages out there and is accepted by library and framework developers and observability tool vendors across the globe. To better understand, let’s look at its core concepts first.
Spans
A Span in OpenTelemetry represents a unit of work or operation and tracks the specific operations performed during a request. It contains various data such as name, time-related data, structured log messages, and other metadata, called attributes, to provide information about the operation it tracks.
Traces
Tracing refers to tracking the path of requests as they navigate through distributed systems. A Trace is made of one or more Spans that facilitate easier debugging.
Metrics
Metrics are numerical data aggregated over a while to provide information about your system or its infrastructure. Examples of Metrics include the system error rate, CPU utilization, and the request rate for a particular service.
Logs
Logs are timestamped messages that are generated by different parts of a system. Unlike Traces, these messages are not necessarily linked to any specific user request or transaction.
Collector
The OpenTelemetry Collector provides a vendor-neutral approach for receiving, processing, and exporting telemetry data. It removes the need to manage and operate multiple collectors (agents) for enhanced scalability. OTel Collectors also supports various open-source observability data formats like Jaeger, Prometheus, and Fluent Bit while allowing to send data to open-source and commercial observability backends. Instrumentation libraries export their telemetry data to the local Collector agent by default.
OTel with .NET
Microsoft has selected OTel as the official telemetry solution for all their frameworks and class libraries as of the .NET 6 release. It includes ASP.NET, Entity Framework, HTTP Clients, etc and is backward compatible with .NET Standard 2.0.
The built-in namespaces for OTel in .NET are System.Diagnostics.Activity and System.Diagnostics.Metrics.
An OpenTelemetry NuGet package wraps both of the above with an OTel-compliant API for you to use.
Let’s examine how telemetry data is collected with OTel in .NET.
- Instrumenting with Spans: We can use the
System.Diagnostics.Activitynamespace to create Spans in their code. - Tracing Requests: Spans are organized into Traces, allowing the end-to-end tracing of requests across different components or services. The OTel SDK for .NET captures Spans as requests flow through the system, and correlate them to form complete Traces.
- Collecting Metrics: The
System.Diagnostics.Metricsnamespace provides classes and APIs to measure and record various Metrics, such as counts, gauges, histograms, and more. - Logging: We can continue using Logging frameworks such as Serilog or NLog. The Logs generated by different parts of the system (exceptions, warnings, or informational messages), are correlated with the corresponding Spans or Traces. It also provides contextual information during troubleshooting.
- Exporting Data: OTel for .NET integrates seamlessly with the OpenTelemetry Collector. The Collector acts as a vendor-neutral intermediary that receives, processes, and exports telemetry data. Instrumentation libraries in .NET export their collected data to the local Collector agent by default. The Collector supports various open-source observability data formats like Jaeger, Prometheus, and Fluent Bit, allowing developers to send their telemetry data to different observability platforms or backends.
Understanding the Flow of Context
Suppose you are building an Asp .NET Core Web API that uses Entity Framework and an HTTP Client. If you get a request, Asp.NET Core creates a new Trace for that. If your API executes a database call via Entity Framework, it creates a Span for that. Then the Span for the database call connects with the Span created by the Asp.NET Core invocation.
As you can see in the following example, OTel flows context across the application across Spans, connecting the entire flow.
{
"resource": {
"service.name": "example-dotnet-service"
},
"spans": [
{
"name": "HTTP Request",
"kind": "SERVER",
"start_time": "2023-06-07T10:00:00Z",
"end_time": "2023-06-07T10:01:00Z",
"attributes": {
"http.method": "GET",
"http.url": "http://example.com",
"http.status_code": 200
}
},
{
"name": "AspNetCore.Request",
"kind": "INTERNAL",
"parent_span_id": "<ID of HTTP Request span>",
"start_time": "2023-06-07T10:00:05Z",
"end_time": "2023-06-07T10:00:50Z",
"attributes": {
"aspnetcore.route": "/api/users",
"aspnetcore.controller": "UserController",
"aspnetcore.action": "GetAll"
}
},
{
"name": "EntityFramework.DatabaseCall",
"kind": "INTERNAL",
"parent_span_id": "<ID of AspNetCore.Request span>",
"start_time": "2023-06-07T10:00:10Z",
"end_time": "2023-06-07T10:00:15Z",
"attributes": {
"db.system": "SQL Server",
"db.statement": "SELECT * FROM Users"
}
}
]
}
1. “HTTP Request” Span:
- Acts as the parent span.
- Represents the overall HTTP request handling process.
- Contains the entire duration of the request.
2. “AspNetCore.Request” Span:
- Represents the ASP.NET Core request handling.
- Nested within the “HTTP Request” span.
- Contains the start and end events of the ASP.NET Core request.
3. “EntityFramework.DatabaseCall” Span:
- Represents the database calls made by Entity Framework.
- Nested within the “AspNetCore.Request” Span.
- Contains the database call event.
Tracing Pipeline
All Traces of these Activity Sources are built into these libraries, then come to the Tracing Pipeline. They are processed by different processes where they are sampled and filtered out Traces that you need. After processing, the exporter pushes the data to an observability backend.
E2E OTel-Observability and Debugging Insights
Although you can implement your custom Loggers, Metrics, and Traces, it doesn’t always make sense to do it manually. Also, the observability capabilities become limited without using a proper observability backend.
This is where tools like Helios come in. Let’s look at the automated functionality provided by Helios to gain a better understanding.
Step 1: Configuring the Trace Provider
You can start by adding the OTel dependencies. First, go to the project’s root folder and run the below commands.
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol dotnet add package OpenTelemetry.Extensions.Hosting --prerelease dotnet add package OpenTelemetry.Instrumentation.AspNetCore --prerelease
It will also install OpenTelemetry. Next, define the Trace provider as follows.
public void ConfigureServices(IServiceCollection services)
{
// ...
services.AddOpenTelemetryTracing(
(builder) => builder
.SetResourceBuilder(ResourceBuilder.CreateDefault()
.AddService("<YOUR-SERVICE-NAME-GOES-HERE>")
.AddAttributes(new Dictionary<string, object>
{
["deployment.environment"] = "<ENVIRONMENT-NAME>"
}))
.AddOtlpExporter(o =>
{
o.Endpoint = new Uri("https://collector.heliosphere.io/traces");
o.Headers = "Authorization=<YOUR HELIOS TOKEN GOES HERE>";
o.Protocol = OtlpExportProtocol.HttpProtobuf;
})
.AddAspNetCoreInstrumentation() // adding instrumentation of incoming web requests
);
}
Step 2: Using instrumentation libraries
You can use instrumentation libraries to generate telemetry data for a particular instrumented library. For example, the instrumentation library for ASP.NET Core will automatically create Spans based on the inbound HTTP requests.
Each instrumentation library is a NuGet package, which can be installed as follows:
dotnet add package OpenTelemetry.Instrumentation.{library-name-or-type}
Then you can then register it when creating the TraceProvider:
public void ConfigureServices(IServiceCollection services)
{
// ...
services.AddOpenTelemetryTracing(
(builder) => builder
.SetResourceBuilder(...)
.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation() // Http client instrumentation for outgoing HTTP calls
.AddOtlpExporter(o => { ... })
);
}
For more information, refer the Helio’s .NET Installation Instructions.
Once the setup is complete and the service is up and running, it will be visible in the Helios application, allowing you to monitor and manage it effectively. This helps developers identify performance bottlenecks and other issues and gain valuable insights into their applications’ performance.
API observability in Helios includes:
A dynamic API catalog is generated based on actual traffic instrumented in your application.
An API dashboard for every single API and other interfaces between services; it includes recent operations (Spans) as well as key metrics, errors, and stats.
An auto-generated API spec for all HTTP interactions, built based on the actual API calls made when running the application.
Step 3: Collecting HTTP request & response payloads
To collect HTTP payloads, the below options must be added for the .AddAspNetCoreInstrumentation() .
// add the following imports
using System.Text.Json;
using Microsoft.AspNetCore.Http.Features;
// configure web requests instrumentation as follows:
.AddAspNetCoreInstrumentation(o =>
{
o.EnrichWithHttpRequest = async (activity, httpRequest) =>
{
// Request payloadif (!httpRequest.Body.CanSeek)
{
httpRequest.EnableBuffering();
}
httpRequest.Body.Position = 0;
var reader = new StreamReader(httpRequest.Body, System.Text.Encoding.UTF8);
var requestBody = await reader.ReadToEndAsync().ConfigureAwait(false);
httpRequest.Body.Position = 0;
activity.SetTag("http.request.headers", JsonSerializer.Serialize(httpRequest.Headers))
activity.SetTag("http.request.body", requestBody);
// Response payloadvar context = httpRequest.HttpContext;
var bufferedResponseStream = new MemoryStream();
var streamResponseBodyFeature = new StreamResponseBodyFeature
bufferedResponseStream
context.Features.Get<IHttpResponseBodyFeature>()!);
context.Features.Set<IHttpResponseBodyFeature>(streamResponseBodyFeature);
context.Response.OnStarting(async () =
{
try
{
if (bufferedResponseStream.TryGetBuffer(out var data))
{
string responseBody = System.Text.Encoding.UTF8.GetString(data.Array!, 0, data.Count);
activity.SetTag("http.response.headers", JsonSerializer.Serialize(context.Response.Headers));
activity.SetTag("http.response.body", responseBody);
}
}
finally
{
var originalStream = streamResponseBodyFeature.PriorFeature?.Stream;
if (originalStream != null)
{
bufferedResponseStream.Position = 0;
await bufferedResponseStream.CopyToAsync(originalStream).ConfigureAwait(false);
}
}
});
};
}
)
Conclusion
Overall, OpenTelemetry has changed the landscape of distributed tracing, establishing a common standard to implement observability in Libraries and frameworks that are readily pluggable for observability backends. Microsoft has adopted OTel as the default way forward with .NET, making it the standard for observability solution implementations.
However, manually setting a complete observability solution for .NET is time-consuming and not optimal. The combination of OpenTelemetry and Helios provides a robust set of tools for observability and debugging, helping developers quickly identify and resolve issues and deliver high-performing .NET applications making the entire process easier to set up.
I hope you better understand the usage of OTel with .NET. Thanks for reading. Cheers!
