Q55 — What is distributed tracing — why is a single log line not enough in microservices?

What a single log line can tell you — and what it cannot

In a monolith, when a request fails or is slow, the entire execution happened in one process. You search the logs for that request's ID, find a stack trace, and the full context — every function call, every DB query, every error — is there. One log stream, one process, one place to look.

In a microservices system, a single user request might touch 10 services: the API gateway, the auth service, the order service, the inventory service, the payment service, the fraud detection service, the notification service, and so on. Each service logs independently. Each service's log stream shows only what happened inside that service.

When the request is slow — say it took 3 seconds — which service was responsible? Service A spent 50ms. Service B spent 2800ms. Service C spent 150ms. But Service A's log line says "call completed in 3000ms" — it's measuring the entire round trip including waiting for B and C to respond. Service B's logs show "query completed in 2200ms" and "RPC completed in 500ms." How do you know that Service B's slow query is what made the user experience slow?

The answer requires stitching together timing information across multiple services for a single request — which is exactly what distributed tracing does.

What is a span?

The fundamental unit of a trace is a span — a named, timed unit of work. Every meaningful operation in your system is a span:

• Handling an incoming HTTP request: span
• Making an outbound RPC call to another service: span
• Executing a database query: span
• Publishing a message to Kafka: span
• Calling an external API: span

Each span records:

• Name: what this unit of work is ("POST /orders", "SELECT orders", "gRPC PaymentService.Charge")
• Start time and duration
• Service name: which service produced this span
• Trace ID: the global identifier for the entire request — the same across all services
• Span ID: this span's unique identifier
• Parent span ID: which span caused this span (establishing the parent-child relationship)
• Status: success or error
• Attributes/tags: contextual data (user_id, order_id, http.status_code, db.statement)

The parent-child relationship is what makes traces powerful. The root span is the initial incoming request. Every downstream call creates a child span. Every downstream call that service makes creates grandchild spans. The result is a tree — the complete execution graph of the request.

How does context propagate across service boundaries?

When Service A makes an HTTP call to Service B, how does Service B know it's part of the same trace as Service A? Context propagation.

Service A injects the trace context into the outgoing request headers:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

This traceparent header (W3C Trace Context standard) encodes the trace ID and parent span ID. Service B extracts this from the incoming request and uses the trace ID to create its own spans under the correct trace, with the correct parent span ID.

This propagation must happen at every service boundary — HTTP calls, gRPC calls, message queue publishes. Modern tracing libraries (OpenTelemetry, Jaeger client, Datadog agent) handle this automatically via instrumentation: they intercept outgoing calls to inject headers and incoming calls to extract them. Application code typically doesn't handle this manually.

When messages flow through a queue (Kafka, SQS), the trace context is embedded in the message headers. The consumer extracts it and continues the trace — making it possible to trace a request that spans a synchronous HTTP call and an asynchronous message publication in one coherent view.

What does a trace look like?

A trace is visualized as a Gantt chart (waterfall diagram). Time goes left to right. Each service's span is a horizontal bar showing when it started and how long it took. Child spans appear nested under their parent.

[API Gateway        ] 0ms ─────────────────────── 850ms
  [Auth Service     ]  5ms ──── 25ms
  [Order Service    ]  30ms ──────────────────── 820ms
    [DB: SELECT     ]  35ms ── 45ms
    [gRPC: Payment  ]  50ms ────────────────── 810ms
      [Payment Svc  ]  55ms ────────────────── 805ms
        [Fraud Check]  60ms ────── 120ms
        [DB: INSERT ]  125ms ─── 140ms
        [External API]  145ms ──────────────── 800ms  ← SLOW

At a glance, you can see: the entire request took 850ms. The API Gateway and Auth Service were fast. The Order Service spans almost the entire time — but only because it was waiting for the Payment Service. The Payment Service was fast until it called an external API, which took 655ms. Root cause: the external API call in the Payment Service.

Without distributed tracing, finding this would require: looking at Order Service logs to see total time was 820ms, then looking at Payment Service logs to see it called an external API, then cross-referencing timestamps to establish the correlation. With tracing, it's a single diagram, immediately obvious.

What is sampling and why is it necessary?

Recording a full trace for every single request is expensive. A trace includes spans from every service the request touches, with attributes attached. At 100,000 RPS with an average of 8 service calls per request, you're generating 800,000 spans per second. Storing all of them is impractical.

Sampling means: only record traces for a fraction of requests — say 1% (head-based sampling) or all requests that exhibit certain properties (tail-based sampling).

Head-based sampling: The first service in the chain decides at the start of the request whether to trace it — usually by random roll (1% chance = trace it). This decision is propagated with the trace context, so all downstream services either all trace or none do. Simple to implement, but you might miss the rare, interesting slow request that wasn't in the 1% sample.

Tail-based sampling: The trace is always collected in full, but the decision to persist it is made at the end, after the full trace is assembled. This lets you keep 100% of slow requests (latency > 2 seconds), 100% of error requests, and 1% of normal requests. Tail-based sampling requires a trace collector that can buffer complete traces before making the persistence decision — more complex, but much more useful because you never miss an anomalous request.

Modern tracing backends (Tempo, Jaeger, Honeycomb) support tail-based sampling via a collector like the OpenTelemetry Collector with sampling rules.

OpenTelemetry — the standard instrumentation layer

Before 2019, every tracing vendor had its own SDK: Jaeger had its own client, Zipkin had its own, Datadog had its own. If you used Jaeger and wanted to switch to Honeycomb, you re-instrumented your entire codebase.

OpenTelemetry (OTel) is the vendor-neutral standard for traces, metrics, and logs. It defines:

• A unified SDK for instrumenting applications (same API regardless of backend)
• A wire protocol (OTLP) for sending telemetry to collectors
• Automatic instrumentation for common libraries (HTTP clients, gRPC, database drivers, Kafka consumers)

With OpenTelemetry, you instrument once and choose your backend (Datadog, Honeycomb, Grafana Tempo, Jaeger) by configuring the exporter. Switching backends requires a config change, not a code change. This is the current industry standard.

The mental model

Why a single log line isn't enough:
  Microservices: one user request → 10 services → 10 log streams
  Log line shows: "request took 3000ms" → which service was slow?
  Without tracing: manually correlate timestamps across 10 log streams
  With tracing: one Gantt chart → slow service immediately visible

Core concepts:
  Trace = entire journey of one request (tree of spans)
  Span = one unit of work: name, start, duration, service, trace_id, parent_id
  Root span = the incoming request
  Child span = every downstream call (RPC, DB, queue publish)

Context propagation:
  traceparent header (W3C standard) carries trace_id + parent_span_id
  Service A injects into outgoing call → Service B extracts and continues trace
  Works across HTTP, gRPC, message queues (via message headers)
  Modern tracing libs (OpenTelemetry) handle this automatically

Visualization: Gantt chart (waterfall)
  Each span = horizontal bar (start → end)
  Child spans nested under parent
  Slow span immediately visible as the longest bar

Sampling (necessary at scale):
  Head-based: random % → simple, may miss rare slow requests
  Tail-based: buffer full trace → keep 100% of errors + slow → complex but better

OpenTelemetry:
  Vendor-neutral SDK for traces + metrics + logs
  Same instrumentation, swap backend via config
  Auto-instrumentation for HTTP, gRPC, DB, Kafka

When traces fail:
  Unsampled requests → no trace → fall back to logs
  Async gaps (fire-and-forget) → trace breaks unless context injected into message

Tracing answers: "which service, which operation, which call is responsible for this slowness/failure?"
Metrics tell you something is wrong.
Logs tell you what happened in one place.
Traces tell you what happened everywhere, in sequence.

Distributed tracing exists because the causal chain of a request is the most important thing you can't see from logs. Logs give you a single witness's account. A trace gives you every witness's account, in chronological order, with timestamps, in one place. At scale, where a single user request touches a dozen services, that's the difference between a 3-minute investigation and a 3-hour one.