Chapter 6 — The Request's Journey

DNS, TCP, TLS, HTTP, Proxies and Caches — One Hop at a Time, With curl

Part 2: Instruments. Prerequisites: Chapters 1–5.

Opening

The question that ends most arguments in a debugging session is: is it the browser, the network, or the server? The browser has extensions, caches, service workers, CORS and cookies. The network has DNS, firewalls, proxies and CDNs. The server has your code. Until you know which one is failing, every theory is a guess.

The way to find out is to take things away. curl is a command-line HTTP client with no extensions, no cache, no service worker and no CORS. If a request fails the same way in curl, the browser is innocent. If you can make each hop — DNS, TCP, TLS, HTTP — succeed or fail on its own, you know exactly where the request dies.

This chapter follows a request from the moment you type a URL to the moment the first byte comes back, and gives you the command that tests each hop.

Part 1: What Each Part of a URL Decides

https://api.example.com:443/v1/orders?status=open
└─┬─┘   └──────┬──────┘ └┬┘ └───┬───┘ └────┬────┘
scheme       host      port   path      query

Each piece controls a different part of the journey:

Notice how much depends on the host. A request to api.example.com and the same request to its IP address 203.0.113.10 can behave completely differently: different certificate, different site on a shared server, different cookies.

Part 2: Hop 1 — DNS

DNS turns a hostname into an IP address. Your machine asks a resolver (usually your ISP's, your company's, or a public one like 8.8.8.8), and the resolver finds the answer from the domain's authoritative name servers.

The records you will meet:

Every answer has a TTL (time to live) — how many seconds resolvers and machines may cache it. This one number explains most DNS confusion.

The commands

dig +short api.example.com              # what does my resolver say?
dig api.example.com                     # full answer, including the TTL
dig @8.8.8.8 +short api.example.com     # what does a different resolver say?
nslookup api.example.com                # the same, available on every OS

The DNS bugs that actually happen

• "I changed DNS and some users still go to the old server." Resolvers cache the old answer until its TTL expires. If the TTL was 86400, some users can see the old IP for up to a day. The fix is planning: lower the TTL a day before a migration, and keep the old server running until the old TTL has passed.
• Different answers in different places. Split-horizon DNS returns internal IPs inside a company network and public IPs outside. A name that works on the VPN may not resolve, or resolve differently, off it.
• A local override. /etc/hosts (or C:\Windows\System32\drivers\etc\hosts) beats DNS on that machine. A forgotten line there makes one laptop go somewhere different from everyone else.
• The browser's own cache. Chrome caches DNS separately; chrome://net-internals/#dns lets you clear it.
• IPv6 that half works. If a host has an AAAA record but IPv6 is broken on the path, clients try IPv6 first and fall back to IPv4 after a delay. The symptom is a consistent extra second or two on the first connection. curl -4 and curl -6 force one or the other.

Part 3: Hop 2 — TCP

With an IP, the client opens a TCP connection to a port. You can test that hop by itself, with no HTTP at all:

nc -vz api.example.com 443
# Connection to api.example.com port 443 [tcp/https] succeeded!

The three outcomes are the three codes from Chapter 3:

Part 4: Hop 3 — TLS

For https, the client and server perform a TLS handshake. The client sends the hostname it wants — SNI (Server Name Indication) — so a server hosting many sites can present the right certificate. Then the client checks that certificate: trusted issuer, not expired, matching hostname.

To see exactly what the server presents:

openssl s_client -connect api.example.com:443 -servername api.example.com

It prints the certificate chain, the issuer, and whether verification succeeded. To check expiry quickly:

echo | openssl s_client -connect api.example.com:443 -servername api.example.com 2>/dev/null \
  | openssl x509 -noout -subject -dates
# subject=CN = api.example.com
# notBefore=Aug  1 00:00:00 2026 GMT
# notAfter=Oct 30 23:59:59 2026 GMT

Two TLS facts that explain strange bugs:

• No SNI, wrong certificate. If a client connects without sending SNI (or connects by IP), the server may present its default certificate, which does not match the hostname.
• Missing intermediate certificate. Browsers often repair a missing intermediate on their own; curl, Node and mobile apps do not. "Works in Chrome, fails from Node" is the signature (Chapter 3).

Part 5: Hop 4 — HTTP, With curl

curl is the single most useful debugging command you will learn. Start from the browser: Network panel → right-click the request → Copy as cURL (Chapter 5). You now have the exact request, outside the browser.

The flags you will use constantly

Be careful with -I. It sends a HEAD request, and some servers and frameworks handle HEAD differently from GET. To see a GET's headers, use -i (or -s -D - -o /dev/null).

Reading curl -v

$ curl -v https://api.example.com/v1/health
* Host api.example.com:443 was resolved.
* IPv4: 203.0.113.10
*   Trying 203.0.113.10:443...
* Connected to api.example.com (203.0.113.10) port 443
* SSL connection using TLSv1.3
* Server certificate:
*  subject: CN=api.example.com
*  expire date: Oct 30 23:59:59 2026 GMT
> GET /v1/health HTTP/2
> Host: api.example.com
> User-Agent: curl/8.7.1
> Accept: */*
>
< HTTP/2 200
< content-type: application/json
< server: nginx
< x-request-id: 7f3c9a
<
{"ok":true}

• Lines starting with * are curl's own notes: DNS result, which IP it connected to, TLS details. The IP here is the one you actually reached — check it is the one you expected.
• Lines starting with > are the request curl sent.
• Lines starting with < are the response headers.

Reading top to bottom, you watch each hop succeed. Wherever the output stops, that is the hop that failed.

Timing each hop

curl -w prints measurements after the request. These numbers are cumulative — each one is measured from the very start.

curl -s -o /dev/null https://api.example.com/v1/orders -w '
dns:        %{time_namelookup}s
connect:    %{time_connect}s
tls:        %{time_appconnect}s
first byte: %{time_starttransfer}s
total:      %{time_total}s
status:     %{http_code}
'

dns:        0.004s
connect:    0.021s
tls:        0.058s
first byte: 1.912s
total:      1.940s

Subtract to get each phase: DNS 4 ms, TCP 17 ms, TLS 37 ms, server think time 1,854 ms (first byte minus TLS), download 28 ms. This is the same breakdown as the DevTools Timing tab (Chapter 5), but from any machine — including from inside your server's network, which tells you whether the slowness is the path or the server.

Comparing hops: bypass layers on purpose

The most powerful move with curl is skipping one layer at a time:

# Through the CDN, as users see it:
curl -si https://www.example.com/pricing | head -20

# Straight to the origin server, skipping the CDN, with the right hostname and SNI:
curl -si --resolve www.example.com:443:198.51.100.7 https://www.example.com/pricing | head -20

# From inside the network, straight to the app, skipping the load balancer:
curl -si http://10.0.3.21:3000/pricing -H 'Host: www.example.com' | head -20

If the origin is correct but the CDN response is wrong, the problem is in the CDN layer — usually caching. If the app is correct but the load balancer response is wrong, the problem is the load balancer. You have bisected the request path (Chapter 1).

Part 6: Proxies, CDNs and Caching

Between the client and your app, responses can be cached — by the browser, a service worker, a CDN, or a reverse proxy. A cached response is fast and cheap. It is also the cause of "I deployed the fix and nothing changed."

Cache-Control — the instructions every cache follows

Reading cache status

Caches report what they did in response headers:

A HIT with a large age after a deploy is the answer to "why don't users see the change?"

The cache key and Vary

A cache stores responses under a cache key — by default roughly the method plus the URL. Two requests with the same key get the same cached response. The Vary response header adds request headers to the key: Vary: Accept-Encoding keeps gzip and brotli versions apart; Vary: Origin keeps CORS responses for different origins apart.

That leads to the most dangerous caching bug there is:

Why "I deployed but it's still old" has three suspects

1. The CDN is still serving a cached copy → check the cache-status header and age; purge (invalidate) the path.
2. The browser cache holds an old file → check DevTools Size column for (disk cache); hashed filenames prevent this for JS and CSS.
3. A service worker is serving old files → Application → Service workers (Chapter 5).

Check them in that order with curl first: if curl (which has no browser cache or service worker) already gets the new version, the CDN is fine and the problem is in the browser.

Part 7: The Headers That Matter in Debugging

The X-Forwarded-Proto header causes a famous bug. A load balancer terminates HTTPS and forwards plain HTTP to your app. Your app sees http, decides to redirect to https, the browser follows, the load balancer forwards as http again — and the browser stops with ERR_TOO_MANY_REDIRECTS. The app must be told to trust the proxy and read X-Forwarded-Proto (in Express: app.set('trust proxy', …)).

Part 8: CORS — Debugged From First Principles

The same-origin policy is the browser's rule that a page may only read responses from its own origin — the exact combination of scheme + host + port. https://app.example.com and https://api.example.com are different origins. So are http://localhost:3000 and http://localhost:8080.

CORS (Cross-Origin Resource Sharing) is how a server says "I allow this other origin to read my responses." It does so with response headers. The browser enforces it; curl and Postman do not, which is why "it works in Postman" tells you nothing about CORS.

Simple versus preflighted requests

Some cross-origin requests are sent directly. Others require the browser to ask permission first with a preflight — an OPTIONS request — and only send the real request if the answer allows it. A request needs a preflight if it uses:

• a method other than GET, HEAD or POST, or
• a custom header such as Authorization, or
• a Content-Type other than form encodings or plain text — including application/json.

So almost every real API call (JSON body, bearer token) is preflighted.

The headers the server must send

Reproducing a preflight with curl

curl -si -X OPTIONS https://api.example.com/v1/orders \
  -H 'Origin: https://app.example.com' \
  -H 'Access-Control-Request-Method: POST' \
  -H 'Access-Control-Request-Headers: authorization, content-type'

Read the response: it must be a 2xx and include Access-Control-Allow-Origin: https://app.example.com and the methods and headers you asked for. Common failures:

• The OPTIONS route returns 401, because auth middleware runs before the CORS handler — preflights never carry credentials.
• The OPTIONS route returns 404 or 405, because no handler exists for it.
• The server returns Allow-Origin: * while the request sends cookies.
• The real request fails with a 500, and the error response skips the CORS headers — so the browser shows a CORS error that hides the real crash (Chapter 4).

Part 9: The Hop-by-Hop Investigation

When a request fails and you do not know where, walk the hops in order. Each step either passes (move on) or fails (you found the layer).

The Worked Example — "The Site Is Down for Some People"

Symptom. After moving the API to a new server last night, some users get timeouts. Others, including you, are fine.

1. Is it the browser? A user sends a curl -v output. It hangs after Trying 192.0.2.40:443... and fails with a connection timeout. The browser is not the cause, and the IP is interesting.

2. DNS. Your dig +short api.example.com returns 203.0.113.10 — the new server. The user's machine is connecting to 192.0.2.40 — the old one.

3. Why different? dig @8.8.8.8 api.example.com shows the record now has a TTL of 86400. Before the change it was also 86400. Resolvers that cached the old answer before the migration will keep it for up to 24 hours.

4. Why a timeout and not an error page? The old server was shut down, and its firewall drops traffic. Dropped packets mean timeout, not refused (Chapter 3).

5. Mitigate. Bring the old IP back temporarily — either restart the old server as a proxy to the new one, or move the old IP to the new machine. Users recover immediately.

6. Prevent. Lower the TTL to 300 seconds at least a day before any migration, migrate, confirm, then raise it again. Keep the old server alive until a full old-TTL period has passed.

The Causal Chains

The Rule You Keep Forever

Take layers away until the failure changes. curl removes the browser, dig tests DNS, nc tests the port, openssl tests TLS, and --resolve skips the CDN — the layer you removed last before the failure disappeared is the one that is broken.

Vocabulary Map

Practice Problems

Problem 1 — Bisect the path. A page shows last week's prices. Your app's database has today's prices. Write the sequence of curl commands (through the CDN, to the origin, to the app) you would run, and what each result would tell you.

Problem 2 — Read the timing. From inside your cloud network, curl -w shows first byte at 80 ms. From your laptop, first byte is at 2.1 s, with TLS finishing at 1.9 s. Where is the time going, and what would you look at next?

Problem 3 — Postman says yes. A teammate says: "The API works perfectly in Postman, so the frontend is broken." The browser shows a CORS error on a PUT with a JSON body. Explain why Postman's result proves little, and give the exact curl command that reproduces what the browser does first.

Problem 4 — The redirect loop. After moving behind a new load balancer, every page gives ERR_TOO_MANY_REDIRECTS. Locally it works. Explain the most likely cause and how you would confirm it with curl -v.

Problem 5 — Plan a migration. You are moving api.example.com to a new provider next Thursday. The current TTL is 86400. Write the DNS plan, with dates and TTL values, that minimises the number of users sent to the old server.

What's Coming Next

curl lets you see one request. In production, you need to see what happened to thousands of requests you were not watching — after the fact. That is what logs are for. Chapter 7 covers what a useful log line contains, how a single request ID ties together lines from many services, and how to search, count and group logs during an incident to find the first failure instead of the loudest one.