Horizontal vs Vertical Scaling
A bigger machine is simpler than more machines. Use it until it stops being enough.
The story
For years Stack Overflow, one of the busiest sites on the internet, ran on a surprisingly short list of hardware. Its engineers wrote about it publicly: around a dozen web servers, and a small number of very large SQL Server machines with enormous amounts of RAM, so the hot data simply lived in memory.
No sharding. No hundred-node cluster. They made a few machines very big and kept the software simple.
Now the opposite case. A startup with 2,000 users runs Kubernetes across three availability zones with autoscaling on every service. Its cloud bill is bigger than its revenue, and every incident means working out which of forty pods did what.
Neither team is wrong about what scaling is. They've made opposite bets on when you pay for complexity. That bet is the whole topic.
The idea
There are two ways to handle more load.
Vertical scaling means giving one machine more CPU, RAM or faster disks. Your code doesn't change. Today's cloud machines go up to hundreds of cores and terabytes of RAM, which is far more than most products ever need.
Horizontal scaling means running more copies and spreading load across them. In principle there's no upper limit, and you get redundancy for free: if one copy dies, the others carry on. But it only works if the copies are interchangeable.
The catch: horizontal scaling needs stateless servers
You can add a second app server only if a request can land on either one and get the same answer. If server A keeps the user's session in its own memory, a request that lands on server B finds no session and the user is logged out.
So "scale out" really means "move the state somewhere shared first": sessions into Redis or a signed cookie, uploads into object storage, scheduled jobs onto one leader or a queue.
Sticky sessions (the load balancer always sends a user to the same server) look like a shortcut. They aren't: when that server dies its users lose their state, and one heavy user can't be spread across servers.
That's why most real systems end up mixed:
The stateless tier scales out because it's easy. The stateful tier, the database, scales up for as long as it can, because spreading data across machines (replicas, then shards) is the expensive, hard-to-undo part. The Database Scaling Ladder is about that.
See what horizontal scaling buys you when a box dies, and what the load balancer has to get right:
The trade-offs
| Vertical (up) | Horizontal (out) | |
|---|---|---|
| Code changes | None | Servers must be stateless. Shared state moves out. |
| Ceiling | The biggest machine you can buy | Effectively none for stateless work |
| Failure | One box is a single point of failure | Lose one of N and the rest carry the load |
| Cost curve | Cheap at first. The top sizes cost a premium per core. | Roughly linear, plus a load balancer and ops overhead |
| Changing size | Often a restart or failover | Add or remove instances live, with autoscaling |
| Ops complexity | Low: one thing to watch | Higher: deploys, discovery, health checks, distributed debugging |
| Best for | Databases and other stateful systems, early-stage products | Stateless web/API tiers, workers, anything spiky |
Two points people get wrong:
- Vertical isn't "not scalable". A database on a large modern machine handles more than most companies will ever need. Moving off it too early is the most common scaling mistake.
- Horizontal isn't automatically highly available. Three copies in the same rack, all reading one config file and all deployed at the same moment, can still fail together.
When to switch
Cheat sheet
| Situation | Choice |
|---|---|
| Web / API servers | Scale out behind a load balancer. Keep them stateless from day one. |
| Background workers | Scale out on queue depth |
| Primary database | Scale up first. Replicas for reads. Shards last. |
| Cache | Scale up a node, then cluster it (consistent hashing) |
| Early-stage product | One big box plus a managed DB beats a cluster you can't debug |
| "We need 99.99%" | You need at least 2 of everything across zones: horizontal for availability, even if load doesn't need it |
| Session state | Redis or a signed cookie, never server memory |
Say it in the interview
"The app tier is stateless, so I'll scale it out behind a load balancer and get redundancy at the same time. The database I'll scale up as far as I can, since a single large primary is simpler and handles our 3,000 writes a second easily. I'd add read replicas when reads outgrow it, and only shard if writes do."
Go deeper
- SD 2 · Chapter 16 — Load Balancing: how the traffic gets spread out once you scale out.
- SD 2 · Chapter 4 — Why Machines Die: why one big box is also one big risk.