Chapter 1: The Speed of Physics

Curriculum: Distributed Systems from First Principles Chapter: 1 of 24 Part: 1 — Physical Foundation Estimated reading time: 75–90 minutes

Opening

This chapter will change how you read every architecture diagram you ever look at. Right now, when you see "add a cache here" or "put this behind a queue," those are probably patterns you've learned to recognise and apply. After this chapter, they'll be conclusions you could derive yourself — because you'll understand the physical constraint that makes them necessary. Every major architectural decision in distributed systems, from why databases have indexes to why CDNs exist to why Redis is fast, is ultimately a response to a small set of numbers. This chapter is those numbers, where they come from, and how to use them as a reasoning tool. You will cite these numbers, by memory, in every system design conversation for the rest of your career.

Part 1: The Problem Your CPU Has

Your CPU is extraordinarily fast. A modern processor executes somewhere between 3 and 5 billion operations per second. That's not a metaphor — it literally performs arithmetic 3–5 billion times every second. Each operation takes roughly 0.3 nanoseconds.

One nanosecond is 0.000000001 seconds. To put it in human terms: if one nanosecond were one second, one second would be 31.7 years.

Now here's the problem. Your CPU can execute instructions at 0.3 nanoseconds per operation, but reading a value from main memory (RAM) takes roughly 100 nanoseconds. That's 300 times slower than the CPU's execution speed. The CPU is like a chef who can plate a dish in 0.3 seconds but has to wait 90 seconds every time they need an ingredient from the pantry. Most of the time, the chef is standing there doing nothing.

This gap between CPU speed and memory speed is called the memory wall, and it's been one of the most important engineering challenges of the last 30 years.

The solution hardware designers came up with is the cache.

The cache is a small amount of very fast memory that lives physically close to the CPU — sometimes on the same piece of silicon. Because it's close and built from faster (and more expensive) materials, reading from it is dramatically faster than reading from RAM. But because it's expensive, there isn't much of it. A typical modern processor has:

• L1 cache: ~64 KB per core. Access time: ~0.5 nanoseconds.
• L2 cache: ~256 KB per core. Access time: ~7 nanoseconds.
• L3 cache: ~8–32 MB shared across cores. Access time: ~20 nanoseconds.
• Main memory (RAM): 8–64 GB. Access time: ~100 nanoseconds.

The numbers matter less than the ratios. L1 cache is 200 times faster than RAM. The CPU checks L1 first, then L2, then L3, then finally goes to RAM. Each miss drops you to a slower tier.

Why should a software engineer care about CPU caches? Because the way you organise data in memory — whether it's sequential or scattered, whether it fits in cache or thrashes it — has a measurable impact on performance. Arrays are faster than linked lists not because of algorithmic complexity, but because arrays are laid out sequentially in memory (cache-friendly), while linked lists scatter their nodes across RAM (cache-hostile). When you iterate an array, the CPU prefetcher notices the sequential pattern and loads ahead. With a linked list, every node dereference is a potential cache miss. This is why, in practice, a flat array of 1000 items can be faster to iterate than a linked list of 50 items in a cache-thrashing access pattern.

You won't need to tune CPU cache behaviour for your SaaS product. But you will need to understand this principle later — specifically when we talk about how databases lay out data on disk, and why columnar storage (used in analytical databases like BigQuery and Redshift) is so much faster for aggregate queries than row storage.

Part 2: The Memory Hierarchy

Let's zoom out from the CPU cache and build the full picture. Between your CPU and the user's request is a hierarchy of storage, each layer progressively slower and larger.

SPEED     LAYER             SIZE          LATENCY
------    ------            ----          -------
Fastest   L1 Cache          64 KB         0.5 ns
  |       L2 Cache          256 KB        7 ns
  |       L3 Cache          8–32 MB       20 ns
  |       RAM               8–64 GB       100 ns
  |       NVMe SSD          500 GB–4 TB   ~100 µs  (100,000 ns)
  |       SATA SSD          500 GB–4 TB   ~500 µs  (500,000 ns)
  |       Network (same DC) unlimited     ~500 µs  (500,000 ns)
  |       HDD               1–20 TB       ~10 ms   (10,000,000 ns)
Slowest   Network (global)  unlimited     ~150 ms  (150,000,000 ns)

Notice where the same-datacenter network sits: level with a SATA SSD and faster than a spinning disk. Reading a value from another machine's RAM across the datacenter is quicker than one HDD seek on your own machine — which is why a remote cache can beat a local disk.

Notice the jumps. Going from L1 cache to RAM is a 200x slowdown. Going from RAM to SSD is a 1000x slowdown. Going from SSD to a spinning hard drive is roughly another 20x slowdown. And going from a local network call to a cross-continental one is a 300x jump.

These are not small differences. They are civilisation-scale differences. To make them feel real, let's scale everything up so that one L1 cache access equals one second of human time.

L1 cache access    = 1 second
L2 cache access    = 14 seconds
L3 cache access    = 40 seconds
RAM access         = 3.5 minutes
NVMe SSD read      = ~2.8 days
SATA SSD read      = ~14 days
HDD seek           = ~4.5 months
Same-DC network    = ~14 days
Cross-continent    = ~15 years

When your Next.js API route fetches a row from a database on the other side of the world without a cache, you're making a request that, on the CPU's subjective timescale, takes fifteen years. The CPU issues the read, then sits completely idle for the equivalent of fifteen years of human time, then processes the result.

This is why performance engineers sound like they're obsessed with "avoiding round trips." Each network round trip isn't just a small delay — it's an eternity from the computer's perspective.

Part 3: The Numbers You Must Know Cold

There is a famous list of latency numbers attributed to Jeff Dean, a legendary engineer at Google who designed some of the most important distributed systems ever built. The numbers have been updated over the years as hardware has improved, but the ratios have remained roughly stable. These are the numbers that underlie every trade-off decision in distributed systems.

Memorise these. Not as trivia — as a working tool.

A few things to notice here.

First, sequential reads are dramatically faster than random reads, on both SSD and HDD. Reading 1 MB sequentially from an SSD takes 1 ms. A single random read from the same SSD takes 0.1 ms, but you'd need 1000 of them to read the same 1 MB — that's 100 ms total, 100× slower. This explains why databases try to lay out related data sequentially (we'll see this in detail in Chapter 2), and it's the entire reason LSM-tree databases exist (Chapter 10).

Second, notice that an NVMe SSD random read (100 µs) is roughly comparable to a same-datacenter network round trip (500 µs). They're in the same order of magnitude. This means: if you're choosing between reading data from a local SSD and fetching it from a service in the same datacenter, they're not wildly different in latency. Both are reasonable. But adding a cross-continental network hop (150 ms) is catastrophically expensive in comparison — 300 times slower than a local network call.

Third, RAM is fast enough that operations purely in memory feel instant to users. 100 ns × 1000 operations = 0.1 ms. A Redis lookup that hits RAM and returns over a local network takes roughly 500 µs total — imperceptible to a human. This is why in-memory caches are so effective. The data isn't computed faster; it just avoids the disk and (sometimes) the network.

Part 4: The Disk Is a Different World

Let's spend more time on disk, because it's where most of the interesting problems in databases come from.

A traditional hard disk drive (HDD) is a physical spinning platter with a magnetic read head. When your operating system asks for data from a specific location on disk, the platter has to spin until that location is under the read head. This mechanical movement is called a seek, and it takes roughly 10 milliseconds. After the seek, the data passes under the read head at roughly 100–200 MB/s.

10 milliseconds sounds trivial until you scale it. A server-grade HDD can perform roughly 100–200 random seeks per second. That means if your database needs to perform 500 random reads per second, you need 3–5 hard drives just to keep up with the I/O — and that's before doing any computation.

Solid-state drives (SSDs) eliminate the mechanical movement. There's no platter, no seek — just electrical signals. A random read on an NVMe SSD takes roughly 100 microseconds, which is 100× faster than an HDD seek. This is why the shift from HDD to SSD was so significant for databases: it didn't just make things faster, it changed the economic calculus of what was worth optimising.

But even with SSDs, there's an important distinction between sequential and random I/O. When you read data sequentially (the data you need is physically adjacent on disk), the storage device can prefetch aggressively and read large chunks at once. Sequential I/O on an NVMe SSD reaches 3,000–7,000 MB/s. Random I/O on the same drive might reach 700,000 IOPS (I/O operations per second), but each operation reads only a small amount of data — and at 4KB per operation, that's only about 2,800 MB/s at peak, and much less under mixed workloads.

The ratio of sequential to random performance is smaller on SSDs than on HDDs, but it still exists. And it's the reason that database storage engines make such a big deal about how they lay data out on disk. This will come back in Chapter 2 (how Postgres reads pages) and Chapter 10 (why Cassandra writes sequentially via LSM trees).

One more thing about disk that matters for distributed systems: durability. When you write data to RAM, it disappears the moment the power goes out. When you write to disk, it survives. This distinction — volatile vs. durable storage — is why every serious database has a write-ahead log (WAL): a sequential append-only file on disk that records every change before it's applied. If the database crashes mid-write, it can replay the WAL on restart and recover. We'll come back to WALs in Chapter 5 when we discuss replication.

Part 5: The Network Is the Wild Card

In a single machine, the latency numbers above are fixed by physics and hardware. In a distributed system, the network introduces a new class of problems — and its latency is not just slower, it's unpredictable.

A network packet travels at roughly two-thirds the speed of light through fibre optic cables. The speed of light is about 300,000 km/s, so through fibre it's roughly 200,000 km/s. The distance from Mumbai to San Francisco is about 13,000 km. A signal travelling in a straight line would take 13,000 / 200,000 = 65 ms one way, 130 ms round trip. In practice, with routing, switches, and serialisation overhead, it's 150–200 ms — close to the theoretical minimum. Physics, not software, sets this floor.

Within a single datacenter — machines connected by gigabit ethernet switches — a round trip takes roughly 500 microseconds. This is your "fast network" baseline.

The difference between these two scenarios — 500 µs vs 150 ms — is a factor of 300. This factor matters enormously for architecture decisions.

If your service makes a database call in the same datacenter, that's 500 µs of network latency plus whatever the database computation takes. If you're building a system where each user request triggers 5 such calls sequentially, you're looking at 2.5 ms of pure network latency — fine. But if you accidentally make those calls to a service in another region, that's 5 × 150 ms = 750 ms, and your API feels broken.

This is why the first thing you'll hear experienced distributed systems engineers say is: "How many network round trips does this request make?" Not "how much computation does it do" — round trips. Because computation is cheap and fast, but network latency is the dominant cost.

Networks also lie

Here is the thing that makes distributed systems hard — and that you cannot get from a single-machine programming background: networks are unreliable in ways that RAM and disk are not.

When you read from RAM, you either get the data or the machine crashes. There's no middle state where the RAM read takes 10 seconds instead of 100 nanoseconds, or where the RAM returns successfully but silently returns the wrong value, or where the read "might have succeeded but we're not sure."

Networks do all of these things.

A network packet can be delayed by hundreds of milliseconds for no apparent reason (network congestion, switch buffer overflows). It can be dropped entirely and silently, so the sender never knows if the message arrived. It can arrive out of order — packets 1, 3, 2, 4 instead of 1, 2, 3, 4. And in sufficiently broken environments, it can be duplicated — the receiver gets the same packet twice.

This is not a theoretical concern. These things happen constantly in production at scale. Every distributed system is built on top of this unreliability, and the entire discipline of distributed systems is, in large part, about how to reason about and build around it. We'll encounter this directly in Chapter 4 (why replication is hard), Chapter 6 (the CAP theorem), and Chapter 11 (why consensus is fundamentally difficult).

The fallacies of distributed computing

In 1994, Peter Deutsch at Sun Microsystems wrote down what became known as the "fallacies of distributed computing" — a list of assumptions programmers new to distributed systems make that are all wrong. The first one is: the network is reliable. Others include: latency is zero, bandwidth is infinite, the network is secure, and topology doesn't change.

You now know enough to understand why each of these is wrong. The network is unreliable because packets can be dropped. Latency is not zero because physics. Bandwidth is not infinite because wires have limits. The network is not secure because packets travel through shared infrastructure. And topology changes because machines fail and are replaced constantly.

These aren't exotic edge cases. They're the normal operating conditions of a distributed system running at any meaningful scale.

Part 6: Using These Numbers to Think

Memorising latency numbers isn't the point. The point is using them as a reasoning tool. Here's how.

Reasoning about where data should live

You're building a SaaS product. Each API request needs the current user's subscription status to decide what features they can access. Where should you store this?

Option A: Check the database on every request. Option B: Cache it in Redis. Option C: Cache it in the application server's memory.

With the latency numbers in mind:

• Database check (same datacenter): database query + network round trip. For a simple primary key lookup on a well-indexed table, maybe 1–5 ms total.
• Redis cache hit (same datacenter): ~500 µs network + Redis's in-memory lookup (~100 ns). Total: roughly 500 µs.
• Application server memory: ~100 ns. Essentially free.

Option C is fastest but has a problem: each application server has its own copy, so if you have 10 servers and a user upgrades their subscription, 9 of them will serve stale data until the cache expires. This is a consistency problem — we'll explore it fully in Chapter 9.

Option B (Redis) is a good middle ground: fast, shared across all servers, easier to invalidate. The latency numbers tell you it's about 3–5× faster than a database call, which matters if you're doing it thousands of times per second.

Option A is fine at low scale. At 100 requests/second, 5 ms per call is no problem. At 10,000 requests/second, you need 10,000 / (1000/5) = 50 database connections just for this one query — and databases handle concurrent connections poorly at that scale.

This is the kind of reasoning the numbers enable. Not "cache is faster so always cache" — but "at what scale does the database cost become a problem, and what does caching actually buy us?"

Reasoning about the cost of a network hop

You're designing a checkout flow for your SaaS product. When a user clicks "Purchase," your API needs to: (1) verify their identity, (2) check their cart, (3) charge their card via Stripe, (4) update their subscription status, (5) send a confirmation email. All of these involve network calls.

If these calls are sequential and each takes 50 ms: 5 × 50 ms = 250 ms before you can respond to the user.

That's borderline — users start noticing latency above 200 ms. If Stripe has a slow moment and takes 300 ms, your checkout takes 550 ms and feels sluggish.

Two fixes present themselves from the latency numbers:

1. Parallelise independent calls. Verifying identity and fetching the cart can happen simultaneously, not sequentially. If they each take 50 ms in parallel, together they take 50 ms, not 100 ms.
2. Move slow work off the critical path. Sending the confirmation email doesn't need to happen before you respond to the user. Put it in a queue (Chapter 14), respond immediately, and let a background worker send the email. Now the email's latency is irrelevant to the user's experience.

Both of these insights come directly from understanding network latency as the dominant cost.

Reasoning about scale

You're designing a system that stores 100 million user records. A query to find all users in a specific city returns 50,000 rows. Each row is 500 bytes. How fast can this query run?

Without an index, Postgres will scan every row: 100 million × 500 bytes = 50 GB of data. Reading 50 GB sequentially from an NVMe SSD at 3,000 MB/s = ~17 seconds. That's before any processing. Unacceptable.

With a proper index on the city column, Postgres goes directly to the matching rows. The index lookup is O(log n) — for 100 million rows, that's about 27 comparisons, each hitting the B-Tree. Then it reads only the 50,000 matching rows: 50,000 × 500 bytes = 25 MB, which reads in ~8 ms. The difference between 17 seconds and 8 milliseconds is the index.

This is Chapter 3's territory in detail, but notice: the reasoning is grounded in the latency numbers. You're not just saying "indexes make queries faster" — you're saying "without an index, we read 50 GB from disk, and at the physics of SSD sequential throughput, that's 17 seconds." That's a number you can put in front of a product manager, a CTO, or an interviewer.

Part 7: The Numbers in an Interview

In a FAANG system design interview, you'll be asked to estimate scale — how many requests per second your system needs to handle, how much storage it needs, how much network bandwidth. Interviewers are looking for whether you can reason about these things, not whether you memorise the exact traffic numbers for Twitter.

The latency numbers are your inputs to these estimations. The interviewer doesn't care if your estimates are off by 30%. They care that your reasoning is sound and that you understand the physical constraints.

A typical estimation might go like this: "We need to serve 10 million daily active users, each making 10 requests per day. That's 100 million requests per day, or roughly 1,200 per second. Each request involves one database read. A database read on a well-indexed query takes roughly 5 ms. So 1,200 simultaneous reads × 5 ms = we need the database to handle 6 seconds worth of concurrent work per second, or about 1,200 concurrent queries. That's one very powerful database server or a few smaller ones with read replicas."

That reasoning is worth more in an interview than memorising that Twitter serves X billion requests per day. The numbers you used — 5 ms for a database read — came from this chapter.

A few specific numbers to have ready for estimations:

• A single database server handles roughly 5,000–10,000 simple queries per second (Postgres on good hardware, for simple primary key reads).
• A single Redis instance handles roughly 100,000 operations per second.
• A typical API server handles 1,000–10,000 requests per second depending on what the request does.
• Network bandwidth in a datacenter: 1–10 Gbps per server link.
• A single Kafka partition handles roughly 10 MB/s (reads and writes).

These aren't precise benchmarks — hardware varies, workloads vary. They're ballpark figures for back-of-envelope calculations.

Here are those ballpark numbers in use. Every output shows its working, rounded the way you'd say it out loud in an interview.

Vocabulary Map

Practice Problems

These problems are not meant to be answered quickly. Sit with them. Reason from the physical reality of this chapter, not from patterns you've memorised.

Problem 1 — Failure Analysis

Your SaaS application serves 5,000 API requests per second. Each request makes exactly 3 database calls, each taking 5 ms, executed sequentially. The team decides to move the database from the same datacenter to a different region (cross-continental), adding 150 ms of round-trip latency per call.

What happens to your API response times? Calculate the before and after. Then: at what point does this become a user-facing disaster, and what's the minimum architectural change that would fix it without moving the database back?

Problem 2 — Design from Scratch

You're building a feature that shows a user's "online status" (like a green dot in Slack) to their teammates. The status changes whenever a user opens or closes the app. You have 500,000 active users, and each user has on average 20 teammates.

Using only the latency and throughput numbers from this chapter, reason through: Where would you store the online status data? What would happen if you stored it in a relational database and updated it on every open/close event? What would be different if you stored it in Redis? Calculate the actual numbers — writes per second, reads per second, storage size.

Problem 3 — Synthesis

Three concepts from this chapter: sequential vs random I/O, the memory hierarchy, and network round trips.

A senior engineer tells you: "We should move from a row-oriented database (like Postgres) to a column-oriented database (like BigQuery) for our analytics queries." Connect all three concepts to explain why this advice might be correct for analytics but wrong for the transactional part of your SaaS product. Your answer should make specific reference to what physically happens when each type of database processes a query.

Problem 4 — Go Deeper

This chapter gave you latency numbers for storage and network. What about computation? A modern CPU executes ~3 billion operations per second (3 GHz). A bcrypt password hash (commonly used for user authentication) takes roughly 100 ms on modern hardware.

Given that 100 ms of CPU time at 3 GHz represents ~300 million CPU operations: what does this tell you about why bcrypt is secure against brute-force attacks? And what does it tell you about the maximum rate at which your login endpoint can process requests if you're running on a single server with no caching? What happens to your authentication service at 50 logins per second?

(This problem requires you to reason about CPU as a bottleneck, which isn't explicitly covered in this chapter — but all the tools you need are here.)

Problem 5 — Connect to the Previous Chapter

(There is no previous chapter — this is Chapter 1. So this problem connects forward instead.)

You now have a mental model of the memory hierarchy: L1 → L2 → L3 → RAM → SSD → Network. In Chapter 2, you'll learn how Postgres stores data in 8KB "pages" on disk and reads those pages into memory to process queries.

Before reading Chapter 2: predict what Postgres would have to do — at the level of the hierarchy — when you run SELECT * FROM users WHERE email = 'alice@example.com' with no index on the email column, against a table with 10 million rows. How many "layers" of the hierarchy does it cross? Where are the bottlenecks? Write your prediction down. After Chapter 2, check it.

What's Coming Next

Chapter 2 goes one layer deeper into the database. You now know that reading from SSD is ~100 µs and that sequential reads are far faster than random ones. Chapter 2 will show you exactly how Postgres exploits and suffers from these physical realities — how it organises 8KB pages on disk, why a full table scan is so expensive in terms of actual I/O operations, and how the B-Tree index structure turns a 17-second full scan into an 8-millisecond targeted read. The latency numbers from this chapter will stop being abstract and will attach to concrete operations you can watch happen inside a real database.