How to Deploy the OTel Collector on Kubernetes: Step-by-Step guide

The OTel Collector is the component deployed between instrumented applications and observability backends. It is a key part of the OpenTelemetry framework; with this tool, you can collect, handle, and export telemetry data from various sources. It transforms data to be usable for performance optimization, monitoring, and troubleshooting.

If you're using Kubernetes, deploying the OTEL collector is crucial for effective gathering and analyzing telemetry data. Deploying the OTEL Collector on Kubernetes involves environment setup, Helm installation, and some additional configurations.

Continue reading to see the step-by-step guide on how to deploy the OTEL Collector on Kubernetes.

Key Takeaways

• OpenTelemetry standardizes the collection, flow, and analysis of your telemetry data. OTel generally has three main components: the OpenTelemetry SDK, API, and Collector.
• Kubernetes environments can be monitored using the OpenTelemetry Collector with multiple receivers and processors.
• Horizontally scaling OTel Collector pods is a logical solution when working with higher data volumes.
• Auto-scaling, load balancing, and resource restrictions increase performance levels.

Deploying the OTEL Collector on Kubernetes: Step-by-Step Guide

Step 1: Setting up the Environment

OTel Collector deployment requires a separate Kubernetes namespace to host the collector components. This namespace segregation isolates the Collector, which improves organization and ease of management.

The kubectl command-line tool can be utilized to a Kubernetes namespace for the OTel Collector. A namespace called "otel" can be created using the following command:

kubectl create namespace otel

Step 2: Installing OTEL Collector Using Helm

Helm is a Kubernetes package manager, which simplifies the OTel Collector deployment process. This includes installing the OTel Collector via Helm, adding the OpenTelemetry Helm repository, updating Helm repositories, and confirming the installation.  

The OpenTelemetry community maintains a Helm repository that stores the Otel-collector helm chart. The helm repo add command adds this repository to your Helm configuration. You can access OTel Collector Helm charts for installation by adding the OTEL Helm repository with the alias open-telemetry.

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts

Then, use the following command to install the chart with the release name my-otel-demo:

helm install my-otel-demo open-telemetry/opentelemetry-demo

It's crucial to update Helm after adding the Otel-Collector Helm repository to guarantee that you have access to the most recent charts. The following command can update the Helm repositories and retrieve the most recent chart data from all configured Helm repositories:

helm repo update

Once the repository has been added and updated, you can use Helm to install the OTel Collector onto your Kubernetes cluster.

Pro Tip

For optimal functionality, use Helm chart version 0.11.0 or higher.

Typically the release name, chart name, and namespace where the OTel Collector is installed are specified in the installation command.

Using the OTel-Collector Helm chart opentelemetry-collector from the open-telemetry repository, the program will install the OTel Collector into the otel Kubernetes namespace. The installation is designated with the release name otel-collector.

helm install otel-collector open-telemetry/opentelemetry-collector -n otel

Checking whether the OTEL Collector works within the Kubernetes cluster can be done once the installation is finished. Use kubectl commands like

• kubectl get pods
• kubectl get services
• kubectl get deployments

to see the current state of the deployed pods, services, and other resources related to OTEL Collector.

Step 3: Installing a Demo App Using Kubectl

Use the following command to install the demo application on your Kubernetes cluster:

kubectl apply --namespace otel-demo -f https://raw.githubusercontent.com/open-telemetry/opentelemetry-demo/main/kubernetes/opentelemetry-demo.yaml

This command creates the necessary Kubernetes resources, such as pods, services, and deployments, and deploys the application by reading the demo-application.yaml file, which provides the required configurations. Once executed, you may use any of the previously mentioned kubectl get commands to check the status of your application.

Step 4: Configuring the OTEL Collector

Configuring the OTEL Collector is crucial to configure the appropriate collection, handling, and exporting of telemetry data. This stage includes defining exporters to deliver the processed data to different backends, configuring processors to change and enhance the data, and setting up receivers to collect data from several sources, depending on your particular use case.

The three primary portions of the OTEL Collector configuration file are receivers, processors, and exporters. The file is usually written in YAML format. Configurations for particular parts in charge of obtaining, processing, and exporting telemetry data are included in each area.

Configuring Receivers

Receivers own the telemetry collection process, and can support one or more of logs, metrics, and traces. They can integrate with a number of different data sources, including prometheus, jaeger, and kafka, to name a few. To use them appropriately, it is crucial to define the protocols and endpoints associated with a particular data source for the receiver to scrape:

receivers:
 jaeger:
   protocols:
     grpc:
       endpoint: "0.0.0.0:14250"

Configuring Processors

The telemetry data that receivers gather is enhanced and modified by processors. They can modify the datasets, enrich them, add features, and remove things like personally identifiable information (PII).

processors:
 attributes:
   actions:
     - key: "environment"
       value: "production"

Configuring Exporters

Exporters send processed telemetry data to different backend systems, like Prometheus, Jaeger, or another instance of the OTEL Collector. The Prometheus exporter will send metrics data to the server operating at the designated endpoint.

exporters:
 prometheus:
   endpoint: "http://prometheus:9090/metrics"

You can customize the OTEL Collector to gather, process, and export telemetry data in your environment by setting up receivers, processors, and exporters according to your organization’s requirements.

This sample setup file we constructed shows how to add an attribute to telemetry data, configure a Jaeger receiver, and export processed data to Prometheus.

receivers:
 jaeger:
   protocols:
     grpc:
       endpoint: "0.0.0.0:14250"
processors:
 attributes:
   actions:
     - key: "environment"
       value: "production"
exporters:
 prometheus:
   endpoint: "http://prometheus:9090/metrics"

Step 5: Deploying Custom Configurations

Custom configurations can be deployed to set up the OpenTelemetry (OTel) Collector. This procedure includes updating the OTel Collector deployment, changing Helm values, and producing a ConfigMap.

To meet the team's unique needs, you can make a ConfigMap using your custom configuration file (Otel-collector-config.yaml). The precise pipelines and settings you want the OTel Collector to employ are contained in this file.  Here’s an Otel-collector-config.yaml example:

kubectl create configmap otel-collector-config --from-file=otel-config.yaml -n otel

You must change the Otel-Collector Helm values to the newly built custom ConfigMap. This step guarantees that the OTel Collector will use your unique configuration when deployed. Here's an example of how your values.yaml might appear:

config:
 existingConfigMap: otel-collector-config

Lastly, use Helm to update your OTel Collector deployment and apply the updated settings. By using this command, you can update your deployment using the values.yaml file's given parameters.

helm upgrade otel-collector open-telemetry/opentelemetry-collector -f values.yaml -n otel    

Step 6: Verifying the Deployment

Ensuring everything functions appropriately after deploying your customized OTEL Collector setup is essential. This step includes verifying that data is being correctly collected and exported, monitoring the status of your pods, and gaining access to logs for troubleshooting.

Checking the Status of OTel Collector Pods

When checking the system’s current status, first ensure the OTel Collector pods are operational. To find out the current state of the pods in the OTEL namespace, execute the command listed below.

kubectl get pods -n otel

Each pod’s status will be listed by using this command. Ensure that the OTEL Collector pods are in running condition by reviewing them, and if any of the pods are not operating, you may need to troubleshoot further (explained below).

Accessing OTel Collector Logs for Troubleshooting

If the deployment encounters any problems, logs can offer important information to help troubleshoot. Look over the logs for failures or warnings that might point to deployment or configuration issues.

This step is crucial to debug and ensure the collector is operating correctly. Use this command while substituting the name of your pod to view the logs of a particular OTEL Collector pod.

kubectl logs  -n otel

Validating Data Collection and Export

Lastly, you must confirm that telemetry data is being successfully collected and exported by the OTel Collector. To accomplish this, ensure that data arrives as expected by verifying with your target backend system, such as Jaeger, Prometheus, or another observability platform.

Here are the procedures for verifying data export and collection:

• Check the Metrics: If you export metrics, ensure your backend system shows and reports them accurately.  
• Trace Verification: Check the tracing data to ensure your tracing system appropriately captures and displays traces.
• Log Data: Verify that log entries are being received and correctly processed if your telemetry configuration includes logs.

Step 7: Integrating OTEL Collector with Applications

To integrate the OTel Collector with your applications, you must instrument them to gather telemetry data and configure them to send it to the OTel Collector. This way, you can ensure that your apps are appropriately monitored and receive all critical information about their behavior and performance.

Instrumenting a Sample Application with OTel SDKs

You must first use the OpenTelemetry SDKs to instrument your application. OpenTelemetry offers SDKs for Python and Java, among other languages. Below is a quick illustration of how to instrument a Python program:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from flask import Flask

# Initialize tracer provider and add a span processor
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)
span_processor = BatchSpanProcessor(ConsoleSpanExporter())
trace.get_tracer_provider().add_span_processor(span_processor)

# Instrument Flask application
app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)

@app.route("/")
def hello():
   with tracer.start_as_current_span("example-request"):
       return "Hello, OpenTelemetry!"

if __name__ == "__main__":
app.run(debug=False)

Configuring the Application to Send Data to the OTel Collector

Next, set up your application to send the OTel Collector all captured telemetry data. This typically includes editing configuration files or modifying environment variables. To set up a Python program to export data to an OTEL Collector, run these commands:

export OTEL_EXPORTER_OTLP_ENDPOINT="http://:4317"
export OTEL_RESOURCE_ATTRIBUTES="service.name=my-sample-app"

Verifying Data Flow from Application to OTel Collector to Backend

Lastly, confirm that the telemetry data flows to your backend system from your application using the OTel Collector. Verify the OTel Collector Logs to ensure your application is actually providing data to the OTel Collector. Here is the command to examine the logs:

kubectl logs  -n otel

Additionally, ensure you properly receive and process data by checking your observability backend. Search for logs or traces related to your application, to confirm it is flowing properly from your cluster.

The OTel collector will now efficiently gather and export telemetry data right from your cluster into the backend of your choice, enabling you to spend time on your observability platform monitoring and gaining new insights into the functionality of your application.

Scaling the OTEL Collector on Kubernetes

Scaling the OTEL Collector on Kubernetes efficiently to handle larger volumes of telemetry data primarily consists of horizontally scaling your OTel Collector pods. Proper scaling is necessary to maintain system responsiveness and handle high data volumes without experiencing performance degradation.

Horizontal Scaling of OTel collector pods

You can add more pods to the OTEL Collector to scale it horizontally and handle more telemetry data. The kubectl scale command can be used to accomplish this. For instance, you would use the following command to scale the OTEL Collector deployment up to three replicas:

kubectl scale deployment otel-collector --replicas=3 -n otel

Best Practices for Scaling OTel Collector

To ensure peak performance and efficient use of resources, the following best OTel Collector scaling practices should be adhered to:

• Load Balancing: Use load balancing to split the load equally among the OTel Collector pods. Kubernetes services can be used to balance traffic.
• Resource Requests and Limits: To avoid resource contention and guarantee that every pod has enough CPU and memory, provide resource demands and limitations in your deployment requirements.
• Auto-Scaling: To automatically change the number of replicas based on CPU or memory utilization, use the Kubernetes Horizontal Pod Autoscaler (HPA).

Monitoring the Performance and Resource usage of the OTel Collector

Monitoring is essential to ensure that the scaled OTel Collector instances are operating as planned and aren't overloading system resources. You can ensure that your OTEL Collector deployment scales successfully and maintains excellent performance and reliability as the volume of telemetry data increases by adhering to these best practices and routinely monitoring the system.

Conclusion

Implementing the OpenTelemetry Collector on Kubernetes improves monitoring and observability. With proper implementation of the instructions in this article, you can configure the OTel Collector to collect, handle, and export telemetry data. This configuration guarantees scalability and flexibility in controlling your observability pipeline and enhances application monitoring and debugging.

FAQs on How to Deploy OTEL Collector On Kubernetes

How does an OTel collector work?

The OTel collector works by receiving telemetry data from various sources. Then, it processes and translates all data into a standard format. This feature makes analysis, storage, and management easier for several backends.

How to deploy the OTel collector in Kubernetes?

Deploying the OpenTelemetry Collector in Kubernetes involves creating a Kubernetes resource along with a ConfigMap for configuration. You'll define the collector's settings, set up appropriate permissions, and use kubectl to apply the configuration. This allows you to centrally collect and manage telemetry data from your Kubernetes applications, with options for scaling and customization based on your specific needs.

How does an OTel exporter work?

The OTel exporter is a component that takes collected telemetry data and sends it to a specific backend or analysis tool. It transforms the data into the format required by the destination system and handles the actual transmission, whether that's to a cloud-based observability platform, a local database, or another monitoring solution.

Sources

• Kubernetes
• Helm
• OpenTelemetry
• Kubernetes
• Helm
• Kubernetes
• OpenTelemetry
• OpenTelemetry
• OpenTelemetry
• Warp
• Uptrace
• DEV
• Dynatrace