Rolling out OpenTelemetry in the right order

OpenTelemetry promises one way to produce traces, metrics and logs, independent of where they end up. The promise holds, but the project is large: specifications, SDKs for a dozen languages, the Collector with hundreds of components, and an operator for Kubernetes. Teams that try to adopt all of it at once tend to stall with half the services instrumented and a pipeline nobody fully understands.

A rollout goes better in a deliberate order: put the pipeline in place first, get broad coverage cheaply, then control volume and consistency.

Put the Collector in the middle

Applications should not send telemetry directly to a vendor. They should send OTLP to an OpenTelemetry Collector that you operate, and the Collector forwards it to one or more backends.

That one layer of indirection pays for itself many times over. Changing backends becomes a Collector configuration change rather than a redeploy of every service. Credentials for the backend live in one place. Sampling, filtering, attribute redaction and enrichment happen centrally, under the platform team's control.

On Kubernetes, the common layout has two tiers:

• An agent per node, run as a DaemonSet. It receives OTLP from pods on that node, adds Kubernetes metadata with the k8sattributes processor and forwards everything onward.
• A gateway, run as a Deployment with several replicas. It does sampling, batching and export.

A minimal gateway configuration:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
 
processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  batch: {}
 
exporters:
  otlphttp/traces:
    endpoint: https://tempo.observability.internal:4318
  prometheusremotewrite:
    endpoint: https://mimir.observability.internal/api/v1/push
 
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/traces]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]

Put memory_limiter first in every pipeline. Without it, a traffic spike makes the Collector run out of memory and drop everything rather than shedding load gracefully. Build a custom Collector distribution with the OpenTelemetry Collector Builder, or use the contrib distribution while you learn which components you need.

Auto-instrumentation first

Hand-written spans are valuable, but they are not where to start. Auto-instrumentation covers inbound HTTP and gRPC requests, outbound calls, database clients and message queues with no code changes, and that alone produces useful distributed traces across most services.

The OpenTelemetry Operator for Kubernetes makes this a platform concern. Define an Instrumentation resource once:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: default
  namespace: checkout
spec:
  exporter:
    endpoint: http://otel-agent.observability:4318
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_always_on

Then a team opts a workload in with a single pod annotation, such as instrumentation.opentelemetry.io/inject-java: "true". Equivalent annotations exist for Python, Node.js, .NET and Go. The operator injects the agent and sets the standard environment variables, including OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_SERVICE_NAME.

Outside Kubernetes the same variables work. Every SDK reads OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES and the OTLP exporter settings, so a golden path can set them once for every service. See Golden paths are an interface, not a template for why that belongs in the platform rather than in each repository.

Add manual spans later, where auto-instrumentation leaves a gap: the business step inside a request handler, a batch job's stages, or a cache decision that explains latency.

Sample at the tail, not the head

Recording every span from every request is expensive and rarely necessary. Head sampling, where the first service decides at random whether to keep a trace, is cheap but blind: it drops slow and failed requests at the same rate as healthy ones.

Tail sampling waits until a trace is complete, then decides. That lets you keep every error and every slow request, and only a fraction of the uneventful ones. The tail_sampling processor in the gateway does this:

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 800
      - name: sample-the-rest
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

A trace is kept if any policy matches. There is one important constraint: every span of a trace must reach the same gateway replica, or the decision is made on fragments. Put a load-balancing tier in front, using the loadbalancing exporter in the agents with routing_key: traceID, so spans are routed consistently by trace ID.

Keep metrics unsampled. They are already aggregated and cheap compared with traces, and the SLO and burn-rate alerts described in Alerting on error budget burn rate depend on complete counts.

Agree on names

Telemetry from fifty services is only useful if the attributes mean the same thing everywhere. The OpenTelemetry semantic conventions define standard names, and instrumentation libraries already follow them for the protocols they cover:

For your own attributes, write a short list in the same style, with a namespace for your organization, and enforce it in review. Set resource attributes such as service name, version and environment from the platform, not by hand. The HTTP conventions were renamed when they stabilized, so older SDK versions may emit http.method instead of http.request.method. Pin SDK versions across services, or use the Collector's transform processor to normalize during migration.

Choosing backends

Because the Collector speaks OTLP, the backend is a replaceable decision. Common open-source combinations include Grafana Tempo or Jaeger for traces, Prometheus or Mimir for metrics, and Loki or OpenSearch for logs. Most commercial observability vendors accept OTLP directly. The Collector can also export to two backends at once, which makes evaluating or migrating to a new one far less risky.

Logs are often the last signal to move. Existing log shippers work fine; the gain from OpenTelemetry logs is correlation, since the SDK can attach trace and span IDs to each record. The filelog receiver lets the node agent collect container logs, so you can retire a separate agent when ready.

Rollout steps

1. Deploy a Collector gateway with memory_limiter, batch and your current backend, and point one service at it.
2. Add the node agent DaemonSet with k8sattributes and route all OTLP traffic through it.
3. Install the OpenTelemetry Operator and enable auto-instrumentation through the golden path, one team at a time.
4. Introduce tail sampling behind a load-balancing tier once trace volume becomes a cost concern.
5. Publish a short attribute convention and set resource attributes from the platform.
6. Add manual spans where traces show unexplained gaps, and bring logs in last.