Join the new Helios product community on Slack!

Golang Distributed Tracing – OpenTelemetry Based Observability

Written by


Subscribe to our Blog

Get the Latest News and Content

OpenTelemetry (OTel in short) is an open-source observability framework that provides a standard set of vendor-agonistic SDKs, APIs, and tools to connect with observability backends. It supports all major programming languages, including Java, Python, Node.js, and Go. 

In short, it lets you collect, export, and analyze telemetry data from your applications and services, including metrics, traces, and logs. Metrics are measurements of your application’s performance, such as CPU usage, memory usage, and response time. Traces are records of how your application’s requests are handled, including which services are called and how long each call takes. Logs are text messages that your application generates, such as error messages and informational messages.

It is gaining popularity as it allows solving observability and troubleshooting challenges in microservices environments through distributed tracing.

However, integrating OTel with Golang is particularly challenging due to several reasons. 

In this article, you will learn about the challenges of OpenTelemetry Golang, installing OTel Go, and how a new approach for compile-time auto-instrumentation makes the process of adding traces to Golang apps much easier with much less friction.

This article is brought by Helios – allowing developers to install OTel-based tracing with visualization and change the way they observe, debug and test microservices in less than 5 minutes.  You can start using it for free.

The challenge with Go OpenTelemetry 

Golang distributed tracing based on OpenTelemetry is specifically challenging. With dynamic languages (e.g. Python or Node.js), we can replace the implementation of a function at runtime using techniques like monkey patching. For languages like Java with intermediate language, it’s possible to manipulate the bytecode using the Java Agent mechanism to support instrumentation. 

However, Go’s strong typing and compiling into machine code makes it difficult to perform the instrumentation dynamically; making runtime changes to compiled machine code is risky and it may be considered a security problem.

How is Golang tracing done today?

Let’s look at the classical way of integrating OTel distributed tracing with Go  

Golang tracing – Step 1 

First, install the OTel Go library using the below command:

go get go.opentelemetry.io/otel

Step 2 – Import OpenTelemtry Go library 

Next, import the OTel Go library and use the otel.Tracer() function to create a tracer and the Start() function to start a new trace span:

import (

    “context”

    “go.opentelemetry.io/otel”

    “go.opentelemetry.io/otel/api/trace”

)

func main() {

    // Create a new trace context

    tracer := otel.Tracer(“my-service”)

    // Creating a new Span

    ctx := context.Background() 

    _, span := tracer.Start(ctx, “main”) 

    defer span.End()

 

Step 3 – Add attributes and events to the span

Then, add attributes and events to the span. You can use the AddEvent() function to record log events, and the SetAttributes() function to add span attributes.

// Record a log event

span.AddEvent(“some work”)

 

// Record an attribute

span.SetAttributes(trace.StringAttribute(“key”, “value”))

 

However, real-world scenarios will require more configuration than the above. For example, the below code snippet is from the OpenTelemetry-Go GitHub repository. It shows the number of configurations you need to instrument a simple HTTP call.

 client := http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}

As you can see, it requires you to make changes to every HTTP call in order to configure the client’s transport and to instrument the native net/http module. The same goes for server-side net/http, which also requires adding code to every HTTP handler, as can be seen here. This requires making code changes to dozens of files and isn’t maintainable.

Simplifying OpenTelemetry Go instrumentation 

But there’s another way.  Let’s look at a unique approach to provide a safer, maintainable, and more understandable way to implement OTel tracing in Golang. 

Unlike in dynamic languages, when it comes to Golang, this approach focuses on integrating OTel at compilation time.

It uses a combination of AST (Abstract Syntax Tree) manipulations and proxy libraries.

  • AST manipulation – Helios utilizes Go’s AST to parse the source code and replace necessary libraries with its telemetry-enabled proxy library counterparts.
  • Proxy libraries – Wraps the original libraries with instrumentation code, keeping the external API the same.

To better understand, let’s consider a Go observability example of making an HTTP request. 

Step 1 – Creating a proxy library

The classical instrumenting approach requires developers to wrap the http.Handler using the manual otelhttp module with OTel. We resolve this issue by creating an open-source proxy library identical to the original one that exposes the same function signature with OTel support.

The below example shows a part of the proxy library created to handle HTTP requests. This proxy library replaces the need of adding the Otel handler for each endpoint – and so defining the HTTP handler remains the same. 

package helioshttp 

import ( 

  “bufio” 

  “context” 

  “io” 

  “io/fs” 

  “net” 

  realHttp “net/http” 

  “net/url” 

  “time” 

 

  “go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp”

)

func Handle(pattern string, handler Handler) {

 handler = otelhttp.NewHandler(handler, pattern)

 realHttp.Handle(pattern, handler)

}

Step 2 – Replacing imports

We provide a compile-time instrumentor, capable of replacing the  ‘imports’ of the original library with its proxy counterpart. It will also update the go.mod file by adding the proxy library (and making sure it’s up to date). The example below will show you how the imports are updated (Actually, this happens at the AST level):

# Default import

import (

  net/http

)

# Replaced import of instrumented library

import (

 http “github.com/helios/go-sdk/proxy-libs/helioshttp”

)

As you can see, this is a pretty simple process to implement OTel based tracing in Go, automatically replacing the files import statements. So, let’s discuss the advantages of this Golang – OpenTelemetry approach compared to classical Go OTel instrumentation.

If you are ready to dip your feet in the water, move to Implement distributed tracing and observability over microservices in Go – docs guidelines 

Advantages of the Go OTel-based approach

1. Platform agnostic 

Unlike other potential approaches, Helios OTel integration is not tightly coupled with the platform since it is added at the code level in compile time. You can use this with any combination of operating systems and system architectures that Go supports. 

It also avoids platform-related issues like handling different CPU architectures since the ‘imports’ are replaced at the AST level instead of the machine code level.

2. Reducing friction

Helios provides transparency for developers to see what the instrumentation does. Since the proxy library code is open-source, you can also go through it and understand how it works. 

For example, classical OTel net/http proxy configuration requires manual wrap each HTTP endpoint with an OpenTelemetry wrapper handler like the below:

import (

    “net/http”

)

 

helloHandler := func(w http.ResponseWriter, req *http.Request) {

  ctx := req.Context()

  span := trace.SpanFromContext(ctx)

  bag := baggage.FromContext(ctx)

  span.AddEvent(“handling this…”,    

  trace.WithAttributes(uk.String(bag.Member(“username”).Value())))

        ,  = io.WriteString(w, “Hello, world!\n”)

}

 

otelHandler := otelhttp.NewHandler(http.HandlerFunc(helloHandler), “Hello”)

 

http.Handle(“/hello”, otelHandler)

err = http.ListenAndServe(“:7777”, nil)

However, if you use Helios, you do not need to worry about this since the Helios proxy library automatically wraps each http.Handle call. After the import replacement is made, developers can click on the “Handle” function and find the instrumented implementation. This is even an advantage in comparison to instrumentation implementations in languages like Node or Java, where it’s quite hard for developers to know what actually happened under the hood.

 

import (

    http “github.com/helios/go-sdk/proxy-libs/helioshttp”

)

helloHandler := func(w http.ResponseWriter, req *http.Request) {

  ctx := req.Context()

  span := trace.SpanFromContext(ctx)

  bag := baggage.FromContext(ctx)

  span.AddEvent(“handling this…”, 

  trace.WithAttributes(uk.String(bag.Member(“username”).Value())))

        ,  = io.WriteString(w, “Hello, world!\n”)

}

 

 

http.Handle(“/hello”, helloHandler)

err = http.ListenAndServe(“:7777”, nil)

Advanced tools that support this instrumentation approach and add visualization

Helios has its own set of internal tools to generate proxy libraries. These tools take care of most of the library generation and maintenance work. 

Proxy library generator

When a proxy library is generated, we need to ensure that all the exported members in the original library are available in the newly created proxy library. Sometimes, there can be hundreds of functions and variable declarations to create, which is obviously time consuming and error prone.

Helios has its generator for this process, which automatically generates the boilerplate for the proxy library based on the original library. So, creators of a new proxy library only have to implement business logic, and there is no risk of missing out on functions or variable declarations.

Here is an example of an original library and its corresponding proxy library generated by the Helios proxy library generator.

// Original Library 

package test_package

import (

    “context”

)

type ExportedType = string

var ExportedVar = 1234

const ExportedConst = “abcd”

func ExportedFunc() {

}

// Helios Generated library

package heliostest_package

import (

    origin_test_package “test_package”

)

type ExportedType = origin_test_package.ExportedType

var ExportedVar = origin_test_package.ExportedVar

const ExportedConst = origin_test_package.ExportedConst

func ExportedFunc() {

    origin_test_package.ExportedFunc()

}

Library interface validator

This validator ensures that the interface of the proxy library matches the original library. It is a crucial step in generating a proxy library since it helps developers check if a new version of the original library is available and what interface changes they need to make in the proxy library.

Tagging of versions

Helios maintains a version tagging technique to maintain the connection between specific versions of the original libraries and the proxy libraries. It ensures that proxy libraries have correct imports at the compile time.

Conclusion

Golang OTel integration is challenging compared to other programming languages like Python, Java, and JavaScript. It requires manual effort and is often time-consuming. Hence, it is essential to look beyond classical instrumentation approaches with Go.

Helios addresses most of the classical challenges when integrating OTel with Golang. In addition, its observability backend solution seamlessly integrates with OTel accelerating the troubleshooting and development process with actionable data and contextual visualization 

The following example shows a trace visualization generated by Helios for a Go service. You can find the complete visualization on the Helios OpenTelemetry Sandbox.

trace visualization for a Go service

You can sign up with Helios free tier from here to try it out.

Related: How to adopt distributed tracing without compromising data privacy

Subscribe to our Blog

Get the Latest News and Content

About Helios

Helios is a dev-first observability platform that helps Dev and Ops teams shorten the time to find and fix issues in distributed applications. Built on OpenTelemetry, Helios provides traces and correlates them with logs and metrics, enabling end-to-end app visibility and faster troubleshooting.

The Author

Helios
Helios

Helios is a dev-first observability platform that helps Dev and Ops teams shorten the time to find and fix issues in distributed applications. Built on OpenTelemetry, Helios provides traces and correlates them with logs and metrics, enabling end-to-end app visibility and faster troubleshooting.

Related Content

Scaling microservices - distributed tracing
Scaling microservices: Challenges, best practices and tools
Scaling the deployment, in order to meet demand or extend capabilities, is a known challenge in many fields, but it’s particularly pertinent when scaling...
Read More
eBPF
What is eBPF?
What is eBPF? eBPF, or Extended Berkeley Packet Filter, is a kernel technology available since Linux 4.4. It lets developers run programs without adding...
Read More
Trace-based alerts with OpenTelemetry instrumentation to reduce MTTR
How we use trace-based alerts to reduce MTTR
Getting our on-call alerts with the complete context covering all observability pillars – traces, metrics, and logs – is a game changer when troubleshooting...
Read More