Q53 — What is structured logging — why does log format matter at scale?

What does a log line look like before you think about it?

The simplest logging is print() — concatenate whatever you know into a string and write it to stdout. In production, this looks like:

code

2024-03-15 14:32:01 ERROR Failed to process order 7821 for user 4492: payment timeout after 5000ms
2024-03-15 14:32:02 INFO  Order 7821 retried
2024-03-15 14:32:03 INFO  Order 7821 succeeded

This is readable to a human looking at one service on one day. Now scale this up: 50 microservices, 200 servers, 100,000 requests per second, each service writing 3–10 log lines per request. You now have tens of millions of log lines per second, stored in a centralized log aggregation system (Elasticsearch, Splunk, Loki, CloudWatch).

Someone files a bug: "the order confirmation for user 9,341,872 is wrong." You need to find all log lines related to that user, across all services, for the last hour. How do you search unstructured log text at this scale?

What breaks with unstructured logs at scale?

Inconsistent formatting across services. Service A logs "user_id=4492", Service B logs "userId: 4492", Service C logs "User 4492". All three refer to the same concept but the strings are different. No automated tool can recognize them as the same field. Writing a query to find all logs for user 4492 across all services requires knowing and accounting for every format variation.

Regex-based parsing is fragile. Log aggregation systems parse unstructured logs using regex patterns — you write a pattern that extracts "user_id" from the string. Every time a developer changes how they format a log message ("Fixed typo in log: user_id → userId"), the regex breaks and the field disappears from the index. Parsing rules become maintenance debt.

No schema, no indexing. Structured query systems work best when fields have names and types. "Search for all errors where latency > 5000ms and user_id = 4492" requires latency to be a named numeric field. In a free-text log line, the log aggregation system must either try to extract it via regex (brittle) or index the entire line as a blob (slow, large, no range queries).

Correlation is impossible without shared keys. When a request touches five services, each service writes log lines with its own format. Tying those lines together — "show me everything that happened for request ID abc-123 across all services" — requires every service to include the same key (request ID, correlation ID, trace ID) in every log line. In unstructured logging, this is a convention, easily forgotten and inconsistently applied.

What is structured logging?

Structured logging means writing logs as machine-readable key-value data from the start — typically JSON — rather than free-form strings.

json

{
  "timestamp": "2024-03-15T14:32:01Z",
  "level": "error",
  "service": "order-service",
  "message": "payment timeout",
  "order_id": 7821,
  "user_id": 4492,
  "latency_ms": 5000,
  "attempt": 1,
  "trace_id": "abc-def-123"
}

Every field has a name and a type. The log aggregation system ingests this as a document with indexed fields, not a blob of text to parse. Now you can query:

user_id:4492 AND level:error — all errors for this user, across all services, instantly indexed
latency_ms:>2000 AND service:payment-service — slow payment operations
trace_id:abc-def-123 — every log line across every service for this specific request

The query is a structured index lookup, not a full-text search over billions of strings.

What does structured logging enable operationally?

Dashboards and alerting on log fields. You can create a dashboard showing the count of level:error grouped by service, the 99th percentile of latency_ms over time, or the distribution of error_code values. These are impossible with unstructured logs unless you parse them first — and parsed logs lag behind because parsing happens asynchronously.

Correlation across services. Every service includes trace_id (a unique identifier for the request, passed through as a header across all service calls) in every log line. Searching for trace_id:abc-def-123 returns all log lines from all services that handled that specific request — the full picture of what happened, across service boundaries.

Automated analysis. Log analysis tools (anomaly detection, error clustering, request flow visualization) work on structured fields. They can group similar errors by error_type, find which user_id values are generating the most errors, or detect when latency_ms for a particular endpoint spikes — because these are first-class queryable fields, not substrings buried in text.

Schema enforcement. In mature systems, services use a shared logging library that enforces the schema — every log line must have timestamp, level, service, trace_id, and message. Optional fields like user_id, order_id, and latency_ms are typed. This makes log output consistent across an entire organization without relying on convention.

What should go in a structured log line?

Always:

timestamp — ISO 8601, UTC, with millisecond precision
level — debug, info, warn, error, fatal
service — which service emitted this
message — human-readable description of what happened
trace_id / correlation_id — the request's unique identifier, for cross-service correlation

Contextually:

Domain identifiers: user_id, order_id, product_id — whatever is relevant to the operation
latency_ms — for any operation with measurable duration
error_code, error_type — structured representation of failures
endpoint, method, status_code — for HTTP handlers
attempt, retry_count — for retry logic

Never:

Secrets, passwords, tokens, PII that shouldn't be in logs
Extremely high-cardinality fields at debug level without filtering — logging every single SQL query's full text at 100k RPS produces unsustainable log volume

The mental model

code

Unstructured logging:
  Free-form string: "ERROR Failed to process order 7821 for user 4492: timeout"
  Human-readable at small scale
  Breaks at scale:
    No consistent field names → can't aggregate across services
    Regex parsing is brittle → format change breaks indexes
    No typed fields → no range queries (latency > 5000ms)
    Correlation is convention-based → easily broken

Structured logging:
  Machine-readable key-value pairs (JSON):
    { "level": "error", "user_id": 4492, "order_id": 7821, 
      "latency_ms": 5000, "trace_id": "abc-123" }
  Every field: named, typed, indexed
  
Enables:
  Exact field queries: user_id:4492 AND level:error
  Range queries: latency_ms:>2000
  Cross-service correlation: trace_id:abc-123 → all lines for one request
  Dashboards on log fields: error rate by service, p99 latency over time
  Automated anomaly detection on structured fields

Always include: timestamp, level, service, message, trace_id
Add contextually: user_id, order_id, latency_ms, error_code, status_code

Log format is not aesthetics — it's the difference between debugging in minutes
and debugging in hours when something breaks at 3 AM across 50 services.

Structured logging exists because at scale, logs are not read by humans scanning text — they're queried by machines on behalf of humans. Machines need structure. Every free-form string you write into a log at 100k RPS is a field you cannot reliably query, a correlation you cannot make, and an incident that takes 3x longer to resolve.