Chapter 18: Observability — Seeing Inside a System You Can't Touch

Curriculum: Distributed Systems from First Principles Chapter: 18 of 24 Part: 8 — Understanding Running Systems Prerequisites: Chapter 1 (latency numbers — the baseline you measure against), Chapter 9 (Redis — how fast in-memory stores enable real-time metrics), Chapter 14 (message queues — how observability pipelines transport telemetry data), Chapter 17 (rate limiting — what you need to observe to know if it's working) Estimated reading time: 75–85 minutes

Opening

You've now built the full machinery of a distributed system: databases that replicate and shard, caches that absorb reads, consensus algorithms that coordinate writes, message queues that decouple producers and consumers, load balancers that distribute traffic, and rate limiters that protect capacity. These systems are composed — a single user request might touch a load balancer, an API gateway rate limiter, a Redis cache, a PostgreSQL primary, a Kafka topic, and three microservices before a response is returned. When that request fails — when it times out, or returns wrong data, or takes 4 seconds instead of 40 milliseconds — how do you find out where the problem is?

You cannot SSH into a production server and watch execution in real time. There are 50 servers, any of which could be involved. The failure might be transient — it happened once to 0.1% of requests, and by the time you look, it's gone. The failure might be distributed — it's not a single server that's broken, it's an interaction between the caching layer and the database under a specific traffic pattern. The failure might not even look like a failure — latency crept from 40ms to 400ms over 3 days, and no single moment was obviously wrong.

This is the observability problem: making the internal state of a system externally knowable. Not "monitoring" in the old sense of watching whether a server is up or down, but deep visibility into what the system is doing, why, and for whom — before, during, and after an incident.

Observability rests on three types of telemetry data, each answering a different question:

• Metrics answer "what is the system doing right now, in aggregate?"
• Traces answer "how did this specific request flow through the system?"
• Logs answer "what exactly happened on this server at this moment?"

The modern practice of observability is using all three together — correlating a spike in a metric dashboard to the traces generated during that spike to the logs those traces produced. This chapter builds each pillar from its underlying data structures, explains the pitfalls that destroy observability systems in production, and shows how they are wired together in real stacks.

Part 1: Metrics — Measuring the System in Aggregate

A metric is a numeric measurement of a system property, recorded over time. The temperature of a server. The number of HTTP requests per second. The 99th percentile response time of the /checkout endpoint. Metrics answer aggregate questions: "what fraction of requests are erroring?" not "which specific request errored?"

The Four Metric Types

Every metrics system — Prometheus, Datadog, CloudWatch, InfluxDB — supports four fundamental types. Understanding what they store at the bit level determines when to use each.

Counter: A monotonically increasing integer. It only goes up (except when the process restarts, when it resets to 0). Counters count events: total HTTP requests served, total bytes received, total errors thrown.

http_requests_total{method="GET", endpoint="/api/users", status="200"}: 1,482,930
http_requests_total{method="GET", endpoint="/api/users", status="500"}:     4,821

Counters are useless raw — "1,482,930 requests total since deployment" tells you nothing about right now. They become useful as rates: the rate of change over time. Prometheus's rate() function computes this: rate(http_requests_total[5m]) divides the counter increase over the last 5 minutes by 5 minutes, yielding requests/second. A counter that jumped from 1,482,000 to 1,482,930 over 5 minutes = (930/300) = 3.1 requests/second.

Gauge: A value that can go up or down — it represents the current state of something. Memory in use, number of active connections, queue depth, CPU utilisation, current number of rate limiter tokens.

redis_connected_clients: 142
memory_used_bytes: 8,453,021,696  (8 GB)
rate_limiter_tokens{api_key="abc123"}: 47

Gauges are meaningful at a point in time. You query them directly: "how many clients are connected right now?" Unlike counters, they don't need rate conversion.

Histogram: Records the distribution of values by counting how many observations fall into predefined buckets. The canonical use case: response time distribution.

http_response_duration_seconds_bucket{le="0.01"}:   45,320   (requests ≤ 10ms)
http_response_duration_seconds_bucket{le="0.05"}:   89,102   (requests ≤ 50ms)
http_response_duration_seconds_bucket{le="0.1"}:    94,221   (requests ≤ 100ms)
http_response_duration_seconds_bucket{le="0.5"}:    98,441   (requests ≤ 500ms)
http_response_duration_seconds_bucket{le="1.0"}:    99,104   (requests ≤ 1000ms)
http_response_duration_seconds_bucket{le="+Inf"}:   99,201   (all requests)
http_response_duration_seconds_sum: 4,831.2          (total seconds across all requests)
http_response_duration_seconds_count: 99,201         (total requests)

From this, you can approximate any percentile. The 99th percentile (P99) is the bucket at which 99% of observations fall below. In the example: 99% of 99,201 = 98,209 requests. The cumulative count crosses 98,209 somewhere in the le="0.5" bucket. So P99 ≈ 500ms. The mean is sum / count = 4,831 / 99,201 ≈ 48.7ms.

Histograms are the right tool for latency, request sizes, and any value whose distribution matters. The mean alone is deceptive: a mean of 50ms with P99 of 4,000ms means most users are happy and 1% are miserable. The histogram reveals this; the mean hides it.

Summary: Similar to histogram but computes percentiles at the client (in the application) rather than at the query layer. Prometheus recommends histograms over summaries because histograms can be aggregated across instances (you can sum bucket counts across 10 servers to get a fleet-wide histogram); summaries cannot.

Labels — Power and Peril

Every metric has a name and a set of labels (also called tags or dimensions) that identify the specific time series. Labels are key=value pairs that slice the metric:

http_requests_total{method="POST", endpoint="/checkout", status="200", region="us-east-1"}
http_requests_total{method="POST", endpoint="/checkout", status="500", region="eu-west-1"}
http_requests_total{method="GET",  endpoint="/users",    status="200", region="us-east-1"}

Each unique combination of label values is a separate time series stored in the metrics database. Labels make metrics expressive — you can ask "what is the error rate for POST requests to /checkout in us-east-1 specifically?" But labels introduce the cardinality problem: if you have 1 million unique user IDs and you label metrics with user_id, you have 1 million time series per metric. A Prometheus cluster that can handle 10 million time series hits that limit with 10 metrics × 1 million users. The metrics database grinds to a halt.

The cardinality rule: never use high-cardinality identifiers as metric labels. Never label metrics with user IDs, request IDs, session tokens, individual IP addresses, or any value with unbounded unique values. Labels should be low-cardinality: endpoint path (20 values), HTTP method (5 values), status code (30 values), region (5 values), service name (50 values). For per-user analysis, use logs and traces, not metrics.

The RED Method and the Four Golden Signals

Faced with a service you've never seen before, two mental models help you choose which metrics to instrument:

RED method (for services that handle requests):

• Rate: requests per second (the throughput)
• Errors: error rate (fraction of requests failing)
• Duration: response time distribution (latency at P50, P95, P99)

These three metrics describe the health of any request-handling service from the user's perspective. If rate drops while errors increase and duration rises, something is wrong. These are the first metrics every service should emit.

USE method (for resources — CPU, memory, disk, network):

• Utilisation: fraction of time the resource is busy (CPU at 80%, disk at 60% capacity)
• Saturation: queue depth or extent to which the resource has extra work it can't service immediately (CPU run-queue length, network interface send queue)
• Errors: error count (disk I/O errors, network packet drops)

USE is applied to physical resources to diagnose resource bottlenecks. A disk at 99% utilisation, 0% saturation means it's busy but not bottlenecked. A disk at 60% utilisation but high saturation (long I/O queue) means requests are piling up — the disk is a bottleneck despite not being "full."

Google published the four golden signals in the SRE book — similar to RED but with one addition:

1. Latency: How long requests take, specifically including the latency of errors (a request that fails in 10ms looks fine in the latency histogram but it failed — don't hide errors in the latency numbers).
2. Traffic: The demand placed on the system — requests/second, bytes/second, messages/second.
3. Errors: The rate of requests that fail, explicitly or implicitly (returns 200 but with wrong data).
4. Saturation: How "full" your service is — the fraction of its capacity currently in use. At 80% saturation, you have headroom. At 99%, you're one traffic spike from degradation.

Part 2: Distributed Tracing — Following a Single Request Through the System

Metrics tell you something is wrong at the aggregate level. Traces tell you exactly what happened to a specific request. This distinction is fundamental — metrics answer "what is the system's P99 latency?" while traces answer "why did this particular request take 3 seconds?"

The Data Structure of a Trace

A trace is a directed acyclic graph of spans. Each span represents a single unit of work — a function call, an HTTP request to a downstream service, a database query, a cache lookup — with a start time, end time, and metadata (service name, operation name, status, tags).

Trace ID: a3f8c1d2e4b6789a

SPAN: API Gateway                          [0ms ─────────────────────────── 847ms]
  SPAN: Auth Service                       [2ms ──── 18ms]
  SPAN: Rate Limiter (Redis)               [19ms ─ 21ms]
  SPAN: User Service                       [22ms ────────────────── 612ms]
    SPAN: DB Query (SELECT user)           [23ms ──── 31ms]
    SPAN: Recommendation Service           [32ms ──────────────── 598ms]
      SPAN: ML Model Inference             [33ms ────────── 211ms]
      SPAN: Redis Cache (user_prefs)       [212ms ─ 214ms]
      SPAN: DB Query (SELECT products)     [215ms ──────────────── 595ms]  ← 380ms anomaly!
  SPAN: Serialization                      [613ms ──── 621ms]
  SPAN: Response compression              [622ms ── 631ms]

This trace immediately reveals that the problem is in the database query for products inside the Recommendation Service. It took 380ms when it should take ~10ms. The API Gateway's 847ms total latency is almost entirely explained by that one slow query. Without the trace, you'd only see "P99 latency is 847ms" — the trace tells you exactly where those 847ms were spent.

Trace ID: A globally unique identifier for the entire trace — one request's journey through the entire system. Typically a 128-bit random value encoded as a hex string. The same trace ID appears in every span generated by that request, in every service.

Span ID: A unique identifier for one span within the trace. Typically a 64-bit random value.

Parent Span ID: The span ID of the caller. The API Gateway's span has no parent (it's the root span). The Auth Service's span has parent = API Gateway's span ID. The DB Query's span has parent = User Service's span ID. This parent-child relationship encodes the call graph.

Context propagation: The trace ID and parent span ID must travel from service to service, typically in HTTP headers:

HTTP Request from API Gateway to User Service:
  traceparent: 00-a3f8c1d2e4b6789a-f1e2d3c4b5a60000-01
                  ^version ^trace_id       ^parent_span_id ^flags

HTTP Request from User Service to Recommendation Service:
  traceparent: 00-a3f8c1d2e4b6789a-89ab1234cdef5678-01
                                    ^new_span_id (User Service's span)

The W3C traceparent header is the modern standard, supported by OpenTelemetry, Jaeger, Zipkin, and all major cloud providers. Each service extracts the trace context from incoming headers, creates a new span (child of the extracted parent span ID), does its work, and injects its span ID into outgoing headers for downstream services to use as their parent.

What a Span Records

Each span captures:

• Operation name: "db.query", "http.request", "cache.get", "ml.inference"
• Service name: which service generated this span
• Start timestamp and end timestamp (microsecond precision)
• Tags / attributes: key-value metadata. For a database span: db.system="postgresql", db.statement="SELECT * FROM products WHERE...", db.rows_returned=142. For an HTTP span: http.method="GET", http.url="/api/products", http.status_code=200.
• Events: timestamped annotations within a span. "Cache miss at T+5ms", "Retry #1 at T+50ms", "Lock acquired at T+120ms".
• Status: OK, ERROR (with error message), or UNSET.
• Links: references to other traces (useful when a span triggers an async job — the job's trace links back to the originating trace).

Sampling — You Can't Trace Everything

At 100,000 requests/second, storing a complete trace for every request is impossible. Traces generate substantial data — a moderately complex trace with 20 spans, each with attributes and events, is ~10KB. At 100,000 requests/second: 100,000 × 10KB = 1 GB/second of trace data. That's 86 TB/day. You need sampling.

Head-based sampling: Decide at the root span (API Gateway, before any downstream calls) whether to trace this request. Sample 1% of all traffic. Simple to implement — the decision is made once and propagated in the trace context. The problem: you're sampling randomly, so interesting requests (the slow ones, the erroring ones) are only captured 1% of the time. The 847ms request above had a 99% chance of not being traced.

Tail-based sampling: Collect spans from every request but make the sampling decision at the end, after the trace is complete. If the trace had errors, trace duration > 1 second, or any span took > 500ms, keep it. Otherwise, drop it. This captures exactly the requests you care about — the slow ones, the failing ones. The problem: you must buffer complete traces before deciding to keep or drop them, which requires trace-aware infrastructure (Jaeger's agent, Honeycomb's Refinery, Grafana Tempo's compactor). More complex but the right approach for production.

Adaptive sampling: Sample 1% of normal requests, 100% of error requests, 100% of requests slower than P99. Automatically adjusts sampling rates based on service health — when a service degrades, increase sampling to capture more data during the incident.

Real systems: Google Dapper (the paper that introduced distributed tracing) samples at ~0.01% globally. Jaeger defaults to 1% probabilistic sampling. Netflix's Edgar and Honeycomb support tail-based sampling. For high-volume systems, the rule is: 1% head-based sampling with 100% capture for errors and anomalies.

Part 3: Logs — The Ground Truth

Logs are the most primitive form of telemetry — sequential records of discrete events, written to disk or shipped to a log aggregation system. They are the most verbose (high storage cost) and the most informative (arbitrary detail about any specific event). When a trace tells you a database query took 380ms, the logs for that query tell you exactly what the query was, what its execution plan was, and whether it waited on a lock.

Structured Logging — Logs as Data, Not Text

The old approach: emit human-readable text.

2024-01-15 10:23:45 ERROR Failed to process payment for order 8821 after 3 retries: connection timeout

This is useful for a human reading a terminal. It's useless for a machine trying to filter, aggregate, or alert on it — parsing freeform text requires regex, breaks when the format changes, and can't be indexed efficiently.

Structured logging emits logs as key-value pairs (usually JSON), where every field is queryable:

{
  "timestamp": "2024-01-15T10:23:45.182Z",
  "level": "ERROR",
  "service": "payment-service",
  "trace_id": "a3f8c1d2e4b6789a",
  "span_id": "89ab1234cdef5678",
  "message": "Payment processing failed",
  "order_id": "8821",
  "attempt": 3,
  "error_type": "ConnectionTimeout",
  "upstream": "stripe-api",
  "duration_ms": 5003,
  "user_id": "u_88291"
}

Now your log aggregation system (Elasticsearch, Loki, CloudWatch Logs) can index every field. You can query: "all ERROR logs from payment-service where upstream=stripe-api in the last hour" and get results in milliseconds. Critically, trace_id is included — you can pivot from a trace to its logs instantly (or from a log to its trace), connecting the three pillars.

Log Levels — Signal Vs. Noise

The right log level discipline keeps logs useful — too verbose and you can't find the signal, too sparse and you miss critical events.

DEBUG: Internal state useful for development. "Cache key: user:42, hit: false, TTL remaining: 0". Never enabled in production for high-traffic services — the volume would be enormous and cost-prohibitive.

INFO: Normal operational events. Service started. Request received. Payment processed. Rate limit check passed. These represent what the system is doing. Should be sparse enough to represent meaningful milestones, not every line of code.

WARN: Unexpected but recoverable situations. "Database connection pool at 90% capacity." "Rate limit for API key approaching threshold." "Upstream responded with 503, retrying." These deserve attention but don't require immediate action.

ERROR: A specific request or operation failed. "Payment for order 8821 failed after 3 retries." "Database write failed: unique constraint violation." Errors should always include full context: what was being attempted, what failed, with what error, for what user/request.

FATAL / CRITICAL: The service cannot continue — it's about to crash or enter an unrecoverable state. "Cannot connect to database on startup after 10 attempts. Shutting down." Very rare in practice; most errors are request-scoped, not service-scoped.

In production: INFO level as the default, WARNING and ERROR always on. Never DEBUG in production high-traffic paths. Log sampling can be applied to high-frequency INFO logs to reduce cost.

Correlation ID — Tying Logs to Requests

Every request must carry a correlation ID (or use the trace ID as the correlation ID) that is included in every log line generated while processing that request. This requires passing the request context through your application — via thread-local storage, Go's context.Context, or Java's MDC (Mapped Diagnostic Context).

Without correlation IDs, logs from concurrent requests interleave and are impossible to untangle:

[WARN]  Rate limit check for user:42 → 47 tokens remaining
[INFO]  Processing order 8821 for user:99
[ERROR] Payment failed for order:9102
[INFO]  Cache hit for user:42's recommendations
[INFO]  Order 8821 confirmed

Which error belongs to which request? Impossible to tell. With correlation IDs:

[WARN]  trace=a3f8 user=42: Rate limit check → 47 tokens remaining
[INFO]  trace=b291 user=99: Processing order 8821
[ERROR] trace=c84a user=55: Payment failed for order:9102
[INFO]  trace=a3f8 user=42: Cache hit for recommendations
[INFO]  trace=b291 user=99: Order 8821 confirmed

Filter by trace=c84a and you get every log line from that specific failed request, across all services, in order.

Part 4: SLIs, SLOs, and Error Budgets — Making Reliability Quantitative

Metrics and traces tell you what the system is doing. SLIs and SLOs translate that raw data into business-meaningful promises about what the system should do.

SLI (Service Level Indicator): A quantitative measurement of a service behaviour that reflects the user's experience. The SLI is a ratio: the number of "good" events divided by total events.

Availability SLI = successful requests / total requests
                 = (requests with status 2xx or 3xx) / (all requests)

Latency SLI = requests completed within 200ms / total requests

Freshness SLI = reads returning data < 1 minute old / total reads

An SLI must be something you can measure — it requires instrumentation. "Users are happy" is not an SLI. "99.5% of requests complete in under 200ms" is an SLI (you can measure it from your histogram).

SLO (Service Level Objective): A target for an SLI over a rolling time window.

Availability SLO:  99.9% of requests succeed over the trailing 30 days
Latency SLO:       95% of requests complete in < 200ms over the trailing 7 days

The SLO is the engineering team's commitment to themselves — not a legal contract (that's an SLA) but an internal standard. If availability drops below 99.9%, the team knows they need to prioritise reliability work. SLOs are chosen based on user impact: what level of reliability do users actually notice? Research shows users notice latency above 100ms and reliability below ~99%.

Error Budget: The amount of unreliability your SLO permits.

Availability SLO: 99.9%
Error budget: 1 - 0.999 = 0.001 = 0.1% of requests can fail.

In a month (30 days × 24 hours × 3600 seconds = 2,592,000 seconds):
  At 100 requests/second: 259,200,000 total requests.
  Error budget: 259,200,000 × 0.001 = 259,200 failed requests allowed.
  Alternatively: 0.1% × 30 days = 43.2 minutes of 100% downtime allowed.

The error budget is a concrete number. Every failed request consumes error budget. If you're at 80% error budget consumed with 10 days left in the month, you have 20% left — you're on track. If you're at 80% consumed with 20 days left, you're on pace to blow the budget. Engineering responses: freeze new feature deployments (they introduce risk), prioritise stability work, on-call escalation.

Error budgets change the conversation between product teams ("ship more features") and reliability teams ("reduce risk"). Product wants to ship features. Each deployment risks incidents. If the error budget is healthy, there's runway to take risks. If the error budget is nearly exhausted, shipping new features threatens to violate the SLO — a concrete, quantifiable reason to slow down.

Alerting on SLOs: The correct alerting approach is SLO-based, not threshold-based. Threshold-based alerting: "alert if error rate > 1%". This alerts even if it's just a brief 10-second spike that consumes 0.001% of the error budget — irrelevant noise. SLO-based alerting uses "burn rate": how fast is the error budget being consumed?

Error budget burn rate = (current error rate) / (1 - SLO)
                       = error rate / error budget rate

SLO: 99.9% → error budget: 0.1%
Current error rate: 2%
Burn rate = 2% / 0.1% = 20×

At 20× burn rate, the monthly error budget is exhausted in: 30 days / 20 = 1.5 days.
ALERT: error budget will be exhausted in 1.5 days.

A burn rate of 1× means you'll exactly exhaust the error budget by end of the window — borderline, worth watching. A burn rate of 2× means you'll exhaust in half the time — page someone. A burn rate of 14.4× means you'll exhaust in 5% of the window (1.5 hours for a 30-day budget) — wake someone up immediately. Google's SRE book defines specific burn rate thresholds and recommends a multi-window, multi-burn-rate alerting strategy: fast burn triggers fast alerts, slow burn triggers slow alerts, so you catch both sudden incidents and slow degradations.

Part 5: The Observability Pipeline — How Telemetry Moves

Metrics, traces, and logs are generated by your application code. They travel through a pipeline before reaching the dashboards and alerts that engineers use. Understanding the pipeline explains failure modes: why metrics disappear when a service crashes, why traces have gaps, why log searches lag by minutes.

Prometheus — Pull-Based Metrics

Prometheus's fundamental design decision is pull: it actively scrapes metrics from application endpoints rather than having applications push metrics to it. Every application exposes a /metrics HTTP endpoint that returns its current metric values in text format:

# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",status="200"} 1482930
http_requests_total{method="GET",status="500"} 4821

# HELP http_response_duration_seconds Response time histogram
# TYPE http_response_duration_seconds histogram
http_response_duration_seconds_bucket{le="0.01"} 45320
http_response_duration_seconds_bucket{le="0.05"} 89102
...

Prometheus scrapes this endpoint every 15 seconds (configurable). It stores the time series in its local TSDB (Time Series Database), a columnar format optimised for sequential writes and range queries. Prometheus's query language, PromQL, is expressive: rate(http_requests_total{status=~"5.."}[5m]) computes the per-second rate of 5xx errors over the last 5 minutes.

The pull model has a key implication: if the application crashes, Prometheus can detect it — the scrape fails and Prometheus marks the target as down. This is free health checking. Push-based systems (like StatsD) don't know the difference between a quiet application and a crashed one.

Prometheus's weakness: Single-node TSDB doesn't scale to millions of time series or years of retention. Solutions: Thanos or Cortex for horizontally scaled Prometheus, Mimir for fully managed scale. These add object storage (S3) for long-term retention and query federation across Prometheus shards.

OpenTelemetry — The Unification Layer

Before 2019, every observability vendor had its own SDK: you'd add the Datadog SDK for traces, the Prometheus client library for metrics, and Logback for logs. Switching vendors required rewriting instrumentation. OpenTelemetry (OTel) is the CNCF project that standardised the telemetry APIs:

• API: The language-specific interface your code calls (tracer.start_span(), meter.create_counter())
• SDK: The implementation that collects, batches, and exports telemetry
• Collector: A standalone binary that receives telemetry from applications, processes it (filtering, sampling, enriching), and exports to any backend

The Collector is powerful: it can tail-sample traces (buffer spans, decide to keep/drop based on trace outcome), add resource attributes (Kubernetes pod name, node name, cluster ID) that the application doesn't know about itself, rate-limit telemetry export to protect backend systems, and fan out to multiple backends simultaneously. OTel is now the default choice for new instrumentation — write OTel once, switch backends without changing code.

Part 6: Alerting — Knowing When Something Is Wrong

Observability is only useful if it generates actionable signal. Alerting is the mechanism that converts metric data into pages and notifications.

The Two Failure Modes of Alerting

Too many alerts (alert fatigue): Every alert requires a human to investigate. If 80% of pages are false alarms — a brief spike that self-resolved, a metric that crossed a threshold due to normal weekly seasonality — engineers learn to ignore alerts. The page fires, they check it, everything is fine, they go back to sleep. Two weeks later, a real incident fires the same alert, and response is slow because it's been crying wolf.

Too few alerts (missing incidents): Alert thresholds set too high miss real degradations. P99 latency creeping from 40ms to 400ms over 3 days never crosses a hard threshold of 1,000ms — it's a real problem that impacts users, but no alert fires.

The resolution: alert on symptoms, not causes. Alert on what users experience (error rate, latency, availability) — the golden signals. Don't alert on causes (CPU at 90%, memory at 85%) unless they're directly correlated with user-visible impact. CPU at 90% might be fine if latency is still 40ms. Alert on latency, not CPU.

Dead Man's Switch — Alerting on Absence

A subtle requirement: alert when telemetry stops. If your metrics pipeline breaks — Prometheus can't scrape, the OTel Collector crashes, the log shipper stops — you lose visibility. If you only alert on metrics being wrong, you won't alert when metrics disappear. A dead man's switch (or "heartbeat alert") fires when a metric stops being emitted:

# Prometheus alerting rule:
alert: MetricsMissing
expr: absent(up{job="payment-service"}) == 1
for: 2m
annotations:
  summary: "Payment service metrics absent for 2+ minutes"

up is a synthetic metric Prometheus creates: 1 if the last scrape succeeded, 0 if it failed. absent() returns 1 if the metric doesn't exist. If the payment service crashes completely, up disappears, absent() fires, and you're paged. Without this, a crashed service would be invisible to Prometheus.

Runbooks — Making Alerts Actionable

Every alert should link to a runbook: a document (typically a wiki page or markdown file in the repo) that answers:

• What does this alert mean?
• What is the likely cause?
• How do I diagnose it? (Which dashboards, which queries, which logs to check)
• What are the common fixes?
• When should I escalate and to whom?

An alert without a runbook is noise. The engineer who receives the page at 3am doesn't know what the alert means, spends 20 minutes figuring it out, fixes it with luck, and doesn't document it. The next engineer pages at 3am and repeats the experience. Runbooks convert tribal knowledge into institutional knowledge. Every alert in a mature team has a runbook. The runbook is updated after every incident that teaches something new.

Part 7: Connecting the Three Pillars — The Observability Workflow

The power of observability comes from correlating all three signals. Here is how a real incident investigation flows:

Step 1 — Alert fires. The SLO-based alert triggers: "checkout service error budget burn rate is 14×. The 30-day error budget will be exhausted in 2 hours." This means right now, 14× more requests are failing than the SLO permits.

Step 2 — Dashboard (metrics). Open the checkout service dashboard. The RED metrics show: request rate is normal (100 req/s), error rate spiked to 14% starting 8 minutes ago (normally < 0.01%), P99 latency jumped from 50ms to 3,200ms. Filter by endpoint: the spike is entirely on POST /checkout. Filter by status code: status="500" spiked, status="200" dropped proportionally.

Step 3 — Trace explorer. Query for erroring traces in the last 10 minutes on POST /checkout. Pick a representative failing trace. The trace shows: API Gateway → Auth Service (OK, 12ms) → Payment Service → Stripe API call (ERROR, 3,200ms, error="connection refused"). The Stripe API call is failing.

Step 4 — Logs. Filter logs for service="payment-service" level="ERROR" trace_id=<trace_id>. The logs show:

{
  "message": "Stripe API request failed",
  "error": "dial tcp 54.239.28.1:443: connect: connection refused",
  "attempt": 1,
  "stripe_endpoint": "https://api.stripe.com/v1/charges",
  "duration_ms": 3204
}

Step 5 — Identify and fix. The evidence is complete: Stripe's API is unreachable from the payment service. Check: is this a network issue? An outage on Stripe's status page? Yes — Stripe reports an incident on their status page. The fix: route to Stripe's backup endpoint, or queue checkouts for retry and show users "payment processing, please wait." The runbook for "Stripe API unreachable" covers exactly this scenario.

Total investigation time with good observability: 8 minutes. Without observability: hours of SSH-ing into servers, reading logs manually, guessing which component failed.

Part 8: What Good Observability Costs — The Trade-Offs

Observability is not free. Every decision has cost:

Metrics storage: Prometheus TSDB is efficient — about 1–2 bytes per sample with compression. At 1 million time series × 4 samples/minute × 1 byte = 4 MB/minute = 5.7 GB/day. Manageable. At 10 million time series (high cardinality): 57 GB/day. Requires tiered storage with downsampling (keep raw data for 15 days, 5-minute aggregates for 1 year).

Trace storage: 10KB per trace × 1% sampling rate × 100,000 req/s = 100MB/s = 8.6 TB/day. Even at 1% sampling, trace storage is substantial. Jaeger and Tempo use columnar storage with compression; real cost is closer to 1–2 TB/day for typical enterprise traffic.

Log storage: The most expensive pillar. Verbose structured logs with 50 fields per request at 100KB/request (including full request/response bodies) at 100,000 req/s = 10 GB/s. In practice, you emit logs for important events only, not every request body. Typical: 1–10 GB/hour for a medium-scale service, depending on log verbosity. Elasticsearch is cost-intensive at scale — Grafana Loki uses a cheaper "logs without indexes on every field" approach to reduce storage costs.

CPU overhead for instrumentation: The OTel SDK adds ~1–5% CPU overhead in high-throughput services. Histograms are more expensive than counters (maintaining bucket counts). Trace span creation and context propagation adds ~5–20 microseconds per span. For most services, this is negligible. For latency-critical paths (Chapter 1's microsecond operations), instrument carefully.

The rule: instrument at the service boundaries (API Gateway, service-to-service calls, database calls) where the cost is trivially small relative to the value. Instrument sparingly within tight loops. Profile the instrumentation overhead if you're in a latency-critical path.

Vocabulary Map

Practice Problems

Problem 1 — Failure Analysis

A team has the following Prometheus alert:

alert: HighCPU
expr: cpu_usage_percent > 80
for: 5m

This alert has been firing an average of 12 times per week for 6 months. Engineers have been silencing it. Last Tuesday, a real incident occurred — the payment service was down for 45 minutes — but the HighCPU alert was silent during the incident (CPU was at 70%, below the threshold, but the process had hit a deadlock and was processing no requests).

Explain precisely: (a) why the HighCPU alert is wrong — what problem does high CPU indicate vs. what the alert is intended to catch, (b) why a deadlock producing 0 requests is invisible to this alert, (c) redesign the alerting strategy using the RED method and SLO-based burn rate alerting that would have caught the Tuesday incident and would fire far fewer false alarms, (d) what absent() alert would detect a completely crashed payment service.

Problem 2 — Design from Scratch

Design the complete observability system for the multi-tenant API from Chapter 17: 10,000 API customers, 500,000 requests/minute peak, 20 application servers, Redis-backed rate limiting, PostgreSQL database. SLO: 99.5% of requests succeed, 99% of requests complete in < 300ms, measured over rolling 30 days.

Specify: (a) which metrics each layer (rate limiter, application server, database) should emit and which type (counter, gauge, histogram) for each — be specific about labels, (b) what trace spans should be created for a single API request and what attributes each span carries, (c) what structured log fields every log line should include, (d) how you'd implement correlation between traces and logs (what to include in each), (e) the SLI formula for both SLOs, (f) what burn rates trigger different severity alerts, (g) the cardinality risk in your metrics design — which labels could explode the time series count and how you'd prevent it.

Problem 3 — Synthesis

Connect observability to Chapter 8's consistent hashing and Chapter 17's distributed rate limiting.

Your rate limiting system uses consistent hashing to shard rate limit state across 10 Redis nodes. Each API key hashes to one specific node. You instrument the following metrics:

rate_limit_checks_total{result="accepted", node="redis-1"}: counter
rate_limit_checks_total{result="rejected", node="redis-1"}: counter
rate_limit_node_latency_seconds{node="redis-1"}: histogram

You observe in Grafana: rate_limit_node_latency_seconds P99 for node="redis-3" is 45ms while all other nodes show < 2ms. rate_limit_checks_total{result="rejected", node="redis-3"} is 3× higher than other nodes.

Propose a diagnosis: what could cause redis-3 specifically to have high latency and high rejection rate? How would you use traces (what span attributes) to confirm which API keys are hashing to redis-3? What additional metric would tell you if redis-3 is memory-saturated? Connect this to Chapter 8: if redis-3 has too many virtual nodes (Chapter 8's virtual node concept), would that explain the skew? What does the solution look like from both a Chapter 8 perspective (rebalancing) and an observability perspective (what metrics tell you the rebalancing worked)?

Problem 4 — Go Deeper

Prometheus's pull model scrapes metrics every 15 seconds. Between scrapes, fast events are invisible. A payment service handles 1,000 requests/second. A bug causes a 200ms window where 100% of requests fail — 200 requests fail before the bug self-corrects.

(a) What is the probability that the Prometheus scrape captures this event at all? (The scrape interval is 15 seconds; the failure window is 0.2 seconds.) (b) If the scrape does capture it, what does the rate(errors[5m]) PromQL query show? The 200 errors are spread across a 5-minute rate window — compute the rate. How does this compare to the rate during the event (100% error rate)?

(c) This is the "scrape interval blindspot" — fast events wash out in rate calculations over longer windows. How do you detect brief but severe spikes? Prometheus's max_over_time() function returns the maximum value observed over a window — would max_over_time(error_rate[5m]) help? What's the limitation of this approach? (d) For what failure scenarios is distributed tracing strictly necessary — where metrics alone are insufficient? Frame your answer in terms of "which questions can't metrics answer?"

Problem 5 — Connect to Previous Chapters

Connect observability to Chapter 12's Raft and Chapter 4's replication.

In a Raft cluster (Chapter 12) with 5 nodes, define the complete set of metrics that would tell you the cluster is healthy and detect each of the following failure modes before they impact users:

For each failure mode below, specify: which metric deviates, in which direction, and what PromQL alerting expression would detect it.

(a) Leader election storm — the cluster is repeatedly electing new leaders (Chapter 12: randomized election timeouts failing to converge). Hint: there's a Raft term counter.

(b) Replication lag — the leader is accepting writes faster than followers can replicate (Chapter 4's lag, Chapter 12's matchIndex). A follower falls 10,000 log entries behind. Before user reads from this follower return stale data at scale.

(c) Quorum loss — 3 of 5 nodes have failed, the cluster has lost quorum and cannot elect a leader (Chapter 12: election requires majority). New writes are being rejected, existing reads may still work from the 2 surviving nodes.

(d) Split brain near-miss — the cluster experienced a network partition (Chapter 6: CAP), elected two leaders, then the partition healed. No data corruption occurred (Raft prevented it via election term safety — Chapter 12), but the metric record should show this happened so the team can investigate the network partition.

For each: what traces or logs would you need alongside the metrics to diagnose root cause, not just detect that something is wrong?

What's Coming Next

You can now build a distributed system and watch it. Chapter 19 closes the loop — you'll take everything from the last 18 chapters and learn how to think through designing a system from scratch, starting from a blank problem statement. System design interviews at FAANG and staff-level roles aren't tests of knowledge — they're tests of structured thinking: how you decompose a vague requirement ("design Twitter") into a system with specific components, where each component choice is justified by the constraints you've identified. Chapter 19 gives you a framework: how to estimate load, how to identify bottlenecks before writing any design, how to select the right storage primitive for the access pattern, how to handle scale, and how to communicate trade-offs clearly. After 18 chapters of building the toolkit, Chapter 19 teaches you to wield it.

Chapter 18 complete. The observability chapter has a specific recommendation for engineers building SaaS: instrument from day one with OpenTelemetry, deploy Grafana + Prometheus + Tempo as a stack (open source, free), and define at least two SLOs (availability and latency) before your first real user. Problem 1 is the most important — redesigning a bad alert into a good one is a skill you'll use every week on the job. Problem 5 ties Raft to observability in a way that appears in senior engineering interviews: "how would you know if your Raft cluster is healthy?" The answer is the specific metrics in Problem 5.