How To Instrument a Go Application using OpenTelemetry: Complete Guide + Best Practices

OpenTelemetry is an open-source framework which provides libraries, agents, and other components for users to measure and collect service and software performance data. It enables development workflows to utilize standardized and interoperable observability pipelines across various platforms, teams, and even programming languages.

Instrumenting Golang with OpenTelemetry provides users with robust tools to generate, collect, and export telemetry data from your Go applications. OTel easily integrates with Golang to manage traces, record operational metrics, and log application performance and behavior. OpenTelemetry uses Go's lightweight SDK to capture all relevant real-time application and performance data, giving you insights into how well your Go program is running.

OTel simplifies monitoring and issue diagnosis, aided by its libraries and tools which seamlessly integrate into your systems. This lets you understand on a deeper level your distributed system's behavior, performance, and resource utilization, helping you to troubleshoot your software to improve system function and reliability.

Continue reading to learn how to instrument a Golang app using OpenTelemetry, along with some best practices to ensure you configure it correctly.

Key Takeaways

• Instrumenting Golang apps allows you to monitor and troubleshoot memory, CPU, and network usage issues.
• The system in question must be observable - in other words, its components must emit log, metric, and trace data.
• Instrumenting Golang apps with OpenTelemetry ensures the system's metrics and traces are captured.
• Go must be installed before integrating OpenTelemetry into its applications.
• Instrument critical software operations via latency, error rate, and application-specific attribute data, especially operations which are prone to slowing or obstructing performance levels.

A Complete Guide on How to Instrument a Golang App Using OpenTelemetry

Instrumenting Golang applications enables the monitoring and troubleshooting of errors related to memory usage, CPU load, and network activity. This aids in detecting errors proactively, allowing for timely interventions and swift fixes that prevent issues from escalating.

Here are the primary reasons to monitor and troubleshoot your Go applications:

• Enhanced Observability - Achieve detailed, real-time insights into application performance and interactions.
• Proactive Troubleshooting - Detect and diagnose issues early on, to avoid negative impact on user experience.
• Performance Optimization - Pinpoint and resolve performance bottlenecks to boost efficiency and scalability.
• Effective Incident Management - Swiftly navigate and resolve issues via comprehensive trace and log data.
• Informed Decisions - Utilize metrics to make well-informed decisions regarding infrastructure and scalability.
• Compliance and Security - Ensure system security and compliance by monitoring and logging system access and modifications.

Side Note

To make a system observable, you must ensure that its code emits trace, log, and metric data. OpenTelemetry allows for this in two primary ways:

1. Code-based solutions via official APIs and SDKs, for most languages (deeper insight and rich telemetry from your application itself)
2. Zero-code solutions typically through environment variables and other language-specific mechanisms, which generates detailed telemetry from the libraries you use and the environment in which your application runs.

Keep reading to learn in detail how to instrument a Go application using OpenTelemetry.

7 Steps on Instrumenting Golang Using OpenTelemetry

Instrumenting Golang apps with OpenTelemetry allows you to capture telemetry data, in particular metrics and traces, from your Golang code. Follow the step-by-step instructions below to instrument your Golang apps using OpenTelemetry.

Note

Auto-instrumentation of the Golang app eliminates the need for developers to manually add tracing or monitoring code to their applications. With this feature, you can add instrumentation to your codebase dynamically.

Step 1: Set up your Project Environment

Before integrating OpenTelemetry into your Go applications, the Go programming language must be appropriately set up on your system. If you haven't installed Go yet, follow the installation instructions in the official Go documentation.

After successfully installing Go, verify its installation to ensure everything is configured correctly. To do this, open a new terminal window and execute the command go version. This command will show your current Go version, confirming that the installation was successful and that your development environment is ready.

Step 2: Create and Launch an HTTP Server

In this step, you'll craft an HTTP server using Go, forming the backbone for integrating OpenTelemetry.

Begin by creating a new file named main.go. In this file, write code to establish a primary HTTP server configured to listen on port 8080. Specifically, your server should handle incoming requests to the /spinwheel URL path.

The server you create should respond to "Hello, OpenTelemetry!" whenever accessed. To start your server, run the command go run main.go from your terminal. Once the server runs, open your web browser and navigate to http://<YOUR IP>:8080 to view the server's response.

This simple server setup is a practical introduction to web server operations in Go and a foundation for further OpenTelemetry implementation. Here’s what the main.go file should look like:

package main

import (
"log"
"net/http"
)

func main() {
http.HandleFunc("/spinwheel", spinwheel)

log.Fatal(http.ListenAndServe(":8080", nil))
}

It's also necessary to configure the behavior of your /spinwheel handler in a separate file named spinwheel.go. In this file, implement a function that generates a random number from 1 to 12, mimicking the action of a spinning wheel of fortune. The function should then return this result to the client. This setup helps modularize your code, allowing for more apparent organization and easier maintenance.

package main

import (
"io"
"log"
"math/rand"
"net/http"
"strconv"
)

func spinwheel(w http.ResponseWriter, r *http.Request) {
roll := 1 + rand.Intn(12)

resp := strconv.Itoa(roll) + "\n"
if _, err := io.WriteString(w, resp); err != nil {
log.Printf("Write failed: %v\n", err)
}
}

To launch the server, execute the command: go run. (note the inclusion of the period). Next, open your web browser to visit http://localhost:8080/spinwheel to observe the server in operation.

Step 3: Add Dependencies to Integrate OpenTelemetry Into the Go App

Now that your HTTP server is running, the next step is to integrate OpenTelemetry into your Go application. You need to add several dependencies, including the following:

• OpenTelemetry SDK
• Standard exporters for metrics and traces
• Instrumentation for the net/http package

Run the following command to install these packages:

go get "go.opentelemetry.io/otel" \"go.opentelemetry.io/otel/exporters/stdout/stdoutmetric" \ "go.opentelemetry.io/otel/exporters/stdout/stdouttrace" \ "go.opentelemetry.io/otel/propagation" \ "go.opentelemetry.io/otel/sdk/metric" \ "go.opentelemetry.io/otel/sdk/resource" \ "go.opentelemetry.io/otel/sdk/trace" \ "go.opentelemetry.io/otel/semconv/v1.24.0" \ "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"

Executing the above command ensures that your project has all the essential libraries for instrumenting Golang applications using OpenTelemetry. This setup provides comprehensive support for tracing capabilities by adding the following:

• Core OpenTelemetry library
• OTLP trace exporter
• Resource SDK
• Trace API

Note

If you’re instrumenting an app, use the OpenTelemetry SDK for your language. You’ll then use the SDK to initialize OpenTelemetry and the API to instrument your code. Doing so will emit telemetry from your app, Golang instrumentation library, or any library you install that also comes with instrumentation.

Step 4: Initialize the OpenTelemetry SDK

After adding the necessary dependencies, set up the OpenTelemetry SDK. This setup is critical for exporting your application's telemetry data, including traces and metrics. Create a new file named otel.go to bootstrap in the OpenTelemetry pipeline.

In the otel.go file, you will define configurations for trace and meter providers and establish a propagator. Below is an example of how the code in otel.go should look, outlining the initialization process for the OpenTelemetry SDK:

package main

import (
// ...other imports...
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
"go.opentelemetry.io/otel/propagation"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.7.0"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func main() {
// Initialize OpenTelemetry SDK
ctx := context.Background()
exporter, err := otlptracegrpc.New(ctx)
handleErr(err, "Failed to create exporter")
openTelemetryURL := attribute.KeyValue{
Key: attribute.Key("opentelemetry.io/schemas"),
Value: attribute.StringValue("1.7.0"),
}

resource, err := resource.New(ctx,
resource.WithAttributes(
semconv.SchemaURL,
openTelemetryURL,
),
)
handleErr(err, "Failed to create resource")

tracerProvider := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exporter),
sdktrace.WithResource(resource),
)

otel.SetTracerProvider(tracerProvider)
otel.SetTextMapPropagator(propagation.TraceContext{})
}

This configuration initiates the OpenTelemetry SDK, configures the OTLP trace exporter, and establishes the relevant resource attributes. Essentially, it ensures that your Go application is ready to collect and export telemetry data efficiently.

Step 5: Instrument the HTTP Server

The main function still needs to be enhanced to gather telemetry data from your HTTP server.go file, which is done by integrating OpenTelemetry instrumentation. This step involves the use of otelhttp middleware to track and collect metrics and traces for HTTP requests automatically.

Enhance your main function in the main.go file by setting up the OpenTelemetry SDK and then applying the ‘otelhttp.NewHandler’ to your server's HTTP handlers. This method wraps your existing HTTP handlers, enabling them to capture detailed telemetry data seamlessly.

Here’s an example of how to modify your code (note that some parts of the code are simplified for clarity):

package main

import (
…
"context"
"errors"
"log"
"net"
"net/http"
"os"
"os/signal"
"time"

"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func main() {
if err := run(); err != nil {
log.Fatalln(err)
}
}

func run() (err error) {
// Handle shutdown scenarios
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
defer stop()

otelShutdown, err := bootstrap_sdk(ctx)
if err != nil {
return
}
defer func() {
err = errors.Join(err, otelShutdown(context.Background()))
}()

// Start HTTP server.
srv := &http.Server{
Addr: ":8080",
BaseContext: func(_ net.Listener) context.Context { return ctx },
ReadTimeout: time.Second,
WriteTimeout: 10 * time.Second,
Handler: newHTTPHandler(),
}
srvErr := make(chan error, 1)
go func() {
srvErr <- srv.ListenAndServe()
}()

// Handle interruptions
select {
case err = <-srvErr:
return
case <-ctx.Done():
stop()
}

err = srv.Shutdown(context.Background())
return
}

func newHTTPHandler() http.Handler {
mux := http.NewServeMux()

handleFunc := func(pattern string, handlerFunc func(http.ResponseWriter, *http.Request)) {
handler := otelhttp.WithRouteTag(pattern, http.HandlerFunc(handlerFunc))
mux.Handle(pattern, handler)
}

handleFunc("/spinwheel", spinwheel)

// Add HTTP instrumentation for the whole server
handler := otelhttp.NewHandler(mux, "/")
return handler
}

This modification fully instruments your HTTP server to record and export telemetry data, improving performance and usage visibility.

Step 6: Enhance Observability through Custom Instrumentation

For more detailed observability of your application, you can introduce custom instrumentation. This advanced step lets you create traces for specific code operations to more precisely monitor and diagnose your application's behavior.

Incorporate the following enhancements into your main.go file to implement this capability:

package main

import (
// ...other imports...
"go.opentelemetry.io/otel/trace"
)

func main() {
// ...other code...
tracer := otel.GetTracerProvider().Tracer("example")
http.Handle("/", otelhttp.NewHandler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, span := tracer.Start(r.Context(), "my-operation")
defer span.End()

fmt.Fprint(w, "Hello, OpenTelemetry!")
}), "/"))

http.ListenAndServe(":8080", nil)
}

This code initiates a new trace span for an operation labeled "my-operation" whenever the root URL is accessed. The span is directly linked to the context of the incoming request, ensuring that all telemetry data remains coherent and accurately represents the operation's execution.

Since the span automatically closes once the operation completes, it encapsulates the lifecycle of this particular action within your server's operations.

This customization makes your application more observable and helps identify your server's performance bottlenecks or workflow issues.

Step 7: Run the Application

Finally, to run your instrumented application, streamline your module dependencies using the command go mod tidy. Next, configure the OTEL_RESOURCE_ATTRIBUTES environment variable to assign resource attributes, such as the service name and version.

Launch your application once your setup is complete by entering ‘go run main.go’ in the terminal. To generate traces, navigate to http://<your IP address>:8080 in your web browser. The traces generated can be accessed and reviewed through your OpenTelemetry collector or your chosen observability platform.

Did You Know?

OpenTelemetry supports popular Golang frameworks like Gin, Echo, and Beego. These frameworks let you add telemetry to Golang applications without significant changes, making monitoring API requests, database calls, and other critical operations far easier.

10 Best Practices for Effective Instrumentation

Effective instrumentation improves data quality and simplifies analysis and interpretation. It lets teams track system behavior, find bottlenecks, and fix issues before users notice.

Here are some best practices for ensuring effective instrumentation with OpenTelemetry in Go:

Best Practice #1: Instrument Key Operations

Identify critical operations in your software, especially those that can slow down or obstruct optimal application performance. Focus on latency, error rates, and application-specific attributes. This method reduces data noise and ensures that instrumentation does not outweigh its benefits, making issue identification easier.

Here are some suggestions to assist you in determining which components of your application to include:

• Instrument the most critical paths first. These critical paths are typically the parts of your application that are most likely to cause issues or bottlenecks, such as:
  • Database queries
  • External API calls
  • Complex business logic

• Instrument code that handles user requests or data processing, as these can cause performance issues
• Start with a small instrumentation set and gradually expand as needed

Best Practice #2: Include Context Propagation

Leverage OpenTelemetry’s context propagation to associate spans with their parent spans. Ensure you properly propagate the context (which carries trace identifiers and other metadata) across service boundaries. Context propagation is needed for synchronous and asynchronous operations like get requests or other external service calls.

Maintaining trace continuity is especially vital in asynchronous operations or instances where a request travels through multiple microservices. Traces remain unbroken by ensuring consistent context propagation, providing a complete picture of a request's journey.

Best Practice #3: Capture Relevant Attributes and Use a Shared Library for Enhanced Telemetry Consistency

Attributes are key-value pairs that provide contextual details within traces, metrics, and logs. This metadata facilitates quicker troubleshooting and effective issue resolution.

A shared attribute library is crucial to achieving telemetry consistency, standardization, and reusability across diverse services. It centralizes the management of known attributes, promoting uniformity and reusability across various applications and services. This centralization also aids in standardizing attribute names, formats, and semantics, ensuring compatibility and streamlining data analysis. Moreover, a shared attribute library minimizes the likelihood of errors and duplications, enhancing maintainability.

With centralized updates, modifications to attributes can be seamlessly propagated throughout all instrumented components, optimizing the overall observability infrastructure.

Best Practice #4: Utilize Sampling

Efficient sampling can help you reduce the data volume and computational overhead associated with metric collection without sacrificing the quality of your insights.

Configure appropriate sampling strategies to control the amount of telemetry data produced. In OpenTelemetry, efficient sampling might involve using:

• Probabilistic
• Rate-limiting
• Custom sampling

Reduced quantity of telemetry data generation can also help prevent performance degradation, cost escalation, and alert fatigue.

Best Practice #5: Use Semantic Conventions

OpenTelemetry's semantic conventions are crucial to achieving system observability and consistency in distributed tracing and metrics. These conventions standardize the naming and structuring of distributed tracing data, including spans, attributes, and events. This standardization ensures that data is formatted and understandable regardless of source or instrumentation tool.

Semantics cover HTTP requests, database operations, and messaging systems in microservices and distributed architectures. These guidelines help developers uniformly describe network interactions, database accesses, and inter-service communications, simplifying Golang OpenTelemetry tracing and monitoring. For example, OpenTelemetry conventions require http.method and http.status_code in every HTTP traffic span.

This consistency improves trace data filtering, searching, and aggregation across services and tools, enabling more accurate root cause analysis and performance optimization.

Best Practice #6: Instrument with Context

Add context to your metrics when instrumenting your application with OpenTelemetry metrics. Context is additional information that can help you understand the circumstances under which a metric was measured. This practice may include information about the environment, the user, or the measured operation. You can better understand your system's behavior and performance by including context in your metrics.

Best Practice #7: Perform Source-Based Data Aggregation

Performing data aggregation at the source can significantly decrease the amount of data being sent across your observability pipeline, and in turn decrease the quantity of data thrown into high-cost indexes. This method saves significant network bandwidth and storage expenses. It also minimizes the computational load involved in processing unaggregated data.

Within the OpenTelemetry framework, consider utilizing the OpenTelemetry SDK to conduct client-side metric aggregation before transmission to the backend.

Best Practice #8: Utilize the OpenTelemetry Collector

Leverage the OpenTelemetry Collector as a versatile tool in your telemetry toolbox. The OTel collector is an independent buffer for your application's telemetry data, enabling the centralized management of its secrets. The Collector allows multiple data exports to multiple destinations and integrates and correlates data from multiple sources with your application's data. This diversity allows for comparative analysis of different observability solutions, with the flexibility to switch or integrate new solutions as needed.

The Collector excels in processing, filtering, and forwarding metrics and traces data in various formats to your chosen backend. In particular, It can enhance data security in sensitive applications by implementing data filtering rules or anonymization to prevent data leaks.

Best Practice #9: Implement Maximum Auto-Instrumentation Initially, Then Dial It Back

Initially, activate as much auto-instrumentation as possible to maximize the data you collect. This approach allows you to determine the utility of the data before making reductions to eliminate non-essential information. Starting with extensive instrumentation ensures that you capture all potentially critical observability data, which might be missed with limited instrumentation.

Over time, as the auto-instrumentation operates, teams can distinguish between valuable data and extraneous details. For instance, if data regarding service health checks proves unnecessary, you can cease tracking this information. By periodically reviewing and selectively reducing your instrumentation, you can maintain essential monitoring without the burden of excessive data collection and transmission.

Best Practice #10: Start with Auto-Instrumentation and Add Manual Instrumentation Later

Leverage OpenTelemetry's automated instrumentation, which performs a lot of heavy lifting. While it only gives bits of specific information, it speeds up workflow time tremendously.

Manual instrumentation can be helpful once you have benefited from auto-instrumentation. You can add manual instrumentation to root cause a service issue, which was originally flagged by automatic instrumentation.

Conclusion

Instrumenting Golang applications using OpenTelemetry provides users with the ability to trace requests, measure metrics, and log events. OpenTelemetry is the future of ensuring Go-based cloud-native applications remain observable. This system gives developers a complete picture of the application, which helps with debugging and performance optimization.

This guide walks you through using OpenTelemetry to collect and export telemetry in Go applications. OpenTelemetry allows you to observe your entire software system, enabling troubleshooting and performance optimization as applications become more distributed.

FAQs on How to Instrument Golang Applications Using OpenTelementry

How to instrument an app with OpenTelemetry?

To instrument an app, install the OpenTelemetry SDK for your language. Then, initialize OpenTelemetry with the OTel SDK and instrument your code using the corresponding API. It will send telemetry from your app and any libraries you install that include instrumentation.

How to use OpenTelemetry in Go?

To use OpenTelemetry in Go, import the OpenTelemetry Go SDK and initialize the tracer. Instrument your code by creating spans and recording metrics. Export the collected telemetry data to your chosen backend.

What is OpenTelemetry auto instrumentation?

OpenTelemetry auto instrumentation uses pre-built libraries or agents to capture and send telemetry data without modifying your application code. It typically measures CPU, memory, request latency and error rates. Though less flexible than manual instrumentation, implementing it is easier and faster.

What is telemetry in Golang?

Telemetry in Golang refers to collecting, transmitting, and analyzing data from a Go application to monitor its performance and behavior. This process includes metrics like CPU and memory usage, tracing for understanding application flows, and logging for detailed event tracking.

List of Sources

• Lumigo
• Uptrace
• OpenTelemetry
• OpenTelemetry
• Elastic
• Sumo Logic
• OpenTelemetry
• Coralogix
• Lumigo
• Last9
• Equinix
• Codesee
• OpenTelementry
• Honeycomb