Q54 — What is the difference between metrics, logs, and traces — when does each fail you?

Why three different signals?

A running distributed system generates an enormous amount of information. You need to answer fundamentally different questions about it:

"Is my system healthy right now?" — you need a dashboard you can glance at
"Why did request X fail at 2:37 PM?" — you need to reconstruct what happened
"This request was slow — which service in the chain made it slow?" — you need to follow the request's journey

Each question needs a different type of data, with different collection costs, different storage requirements, and different query models. Metrics, logs, and traces are each optimized for one of these needs. Understanding where each excels — and where each fails — tells you when you need all three.

Metrics — aggregate numbers over time

A metric is a numeric measurement sampled or counted over time. HTTP request count. Average latency. Error rate. Database connection pool size. CPU usage. Memory consumption.

Metrics are aggregated — you're not storing one data point per event; you're storing a value that represents the state of the system at a point in time. A counter increments every time a request arrives. A gauge records the current queue depth. A histogram buckets latency measurements by range (how many requests took 0–100ms? 100–500ms? >1000ms?).

These aggregates are sampled on a schedule — every 10 seconds, every minute — and stored in a time-series database (Prometheus, InfluxDB, Datadog, CloudWatch). They're cheap: instead of storing one record per request (which might be millions per second), you store one data point per metric per interval. Retention is long — you can look at metrics from 6 months ago without breaking the bank.

What metrics excel at: dashboards, alerting, capacity planning. "Show me HTTP error rate over the last 24 hours." "Alert me when p99 latency exceeds 2 seconds." "How has database CPU grown over the last 3 months?" Metrics answer "how much" and "how often" efficiently and durably.

Where metrics fail you: they have no context. A spike in error rate tells you that something is wrong and approximately when it started. It cannot tell you which requests failed, which users were affected, why a specific request failed, or where in the call chain the failure originated. Metrics are aggregate — the individual event that was a problem has been collapsed into a number. To answer "why," you need logs or traces.

Logs — records of individual events

A log (structured, as per Q53) is a record of something that happened: a specific request was received, a specific query executed, a specific error was thrown, a specific message was published. Unlike metrics, logs are per-event — you get one log line per event, with context attached.

Logs carry: what happened, when it happened, what the system state was at that moment (user ID, order ID, error message, latency, etc.), and what request or operation it belongs to (trace ID).

Logs are expensive at scale. A high-throughput service generating 10 log lines per request at 100,000 RPS produces 1,000,000 log lines per second. Indexed and stored, this is enormous. Log aggregation systems (Elasticsearch, Loki, CloudWatch) handle this, but storage and query costs are orders of magnitude higher than metrics. Logs are typically retained for 30–90 days, not 6–12 months.

What logs excel at: debugging a specific failure. "Show me all log lines for order 7821." "Show me all errors from the payment service between 14:30 and 14:35." Logs let you reconstruct exactly what happened in a specific case. They're the "video footage" to metrics' "crime statistics."

Where logs fail you: they don't show you the request's journey across service boundaries. A log line in the order service shows what happened in the order service. The fact that the order service called the payment service, which called the fraud detection service, which timed out — that chain is invisible unless you manually correlate trace IDs across three separate log queries. Logs tell you what happened in one place, but they don't stitch together what happened everywhere for a single request. That's what traces do.

Logs also fail for performance questions: "why is this endpoint slow on average?" requires aggregating latency over many requests — a metrics question, not a logs question. Scanning millions of log lines to compute an average is inefficient; a pre-aggregated metric is orders of magnitude faster to query.

Traces — the journey of a single request across services

A trace is a record of everything that happened for a single request as it traveled through your system. Every service the request touched, every downstream call it made, every database query it executed, every external API it called — all captured in one coherent tree.

Each unit of work within a trace is called a span. A span has:

A name (what it represents: "HTTP POST /orders", "SELECT * FROM orders", "gRPC call to PaymentService")
A start time and duration
A parent span ID (which span caused this span)
Tags/attributes (user ID, status code, DB query, anything relevant)
Events (notable moments within the span, like a retry or a cache miss)

Spans form a tree: the root span is the incoming HTTP request. It creates child spans for each downstream call. Those calls create their own child spans for their DB queries and RPC calls. The entire tree — assembled from spans reported by every service — is the trace.

What traces excel at: latency attribution and distributed debugging. "This request took 850ms — why?" The trace shows: order-service spent 10ms, payment-service spent 750ms (of which 720ms was in a DB query). The slow DB query is identified immediately. Without a trace, finding the bottleneck would require correlating logs across multiple services and doing mental math on timestamps.

Traces also answer: "which service is responsible for the degradation we see in our p99 latency?" Run a trace sample across recent slow requests, find the service that consistently contributes the most time, and you have your answer.

Where traces fail you: traces are sampled. Instrumenting every single request end-to-end at 100,000 RPS is prohibitively expensive — the tracing infrastructure itself would consume significant resources, and storage would be enormous. Production systems typically sample 1–10% of traces. This means you have complete traces for some requests, but not all. If the specific failing request was not sampled, you don't have a trace for it — you need to fall back to logs.

Traces also don't tell you aggregate health — they're per-request. "What's the error rate?" is a metrics question. "Why did this specific request error?" is a traces question.

How they work together in practice

The three signals are not redundant — they're complementary layers of the same system:

Metrics alert you that something is wrong. Your p99 latency dashboard spikes. Error rate climbs. You know something is wrong, and roughly when it started.
Traces show you where the problem is. You pull a sample of slow traces from the time window. They consistently show payment-service taking >800ms. You've localized the problem to one service.
Logs tell you why. You search logs from payment-service for that time window: level:error AND service:payment-service. You find a log entry: database connection pool exhausted, queries queuing. Root cause identified.

Each signal gets you one layer closer to the answer. Metrics narrow the window and localize the symptom. Traces identify which service and which operation. Logs reveal the specific event and context.

Modern observability platforms (Datadog, Honeycomb, Grafana stack) integrate all three — linking from a metrics spike to the relevant traces, and from a trace to the relevant log lines, with a single click.

The mental model

code

Metrics:
  What: numeric aggregates over time (counters, gauges, histograms)
  Granularity: per interval (every 10s), not per event
  Cost: very low — one data point per metric per interval
  Retention: long (months to years)
  Answers: "how much?" "how often?" "is the system healthy?"
  Use for: dashboards, alerting, capacity planning, SLOs
  Fails at: explaining why a specific request failed (no context, aggregated)

Logs:
  What: records of individual events with context
  Granularity: per event, per log statement
  Cost: high — one record per request × N log lines × request rate
  Retention: medium (30–90 days)
  Answers: "what happened to this specific request/entity?"
  Use for: debugging specific failures, auditing, forensics
  Fails at: cross-service request flow (one service's view only)
             aggregate performance questions (scanning millions of lines)

Traces:
  What: end-to-end journey of one request as a tree of spans
  Granularity: per request (sampled — 1–10% of traffic)
  Cost: medium — spans are small, but sampled
  Retention: short-medium (7–30 days)
  Answers: "where in the call chain is the slowness/failure?"
  Use for: latency attribution, distributed debugging, service dependency mapping
  Fails at: specific unsampled requests (might not have the trace)
             aggregate health (per-request, not population-level)

In practice — the three-step debugging loop:
  1. Metrics alert → something is wrong, roughly when and what signal
  2. Traces localize → which service, which operation is slow/failing
  3. Logs explain → specific error message, context, root cause

They're not alternatives — they're three lenses on the same system,
each revealing what the others cannot.

Metrics tell you the system's vital signs. Logs tell you what each patient said. Traces tell you who talked to whom and in what order. You need all three because each one has a blind spot the others can see.