Q09 — Why do databases exist — why not just use files?

What's wrong with files?

If you want to store data, a file seems like the obvious tool. Write bytes in, read bytes out. Your operating system already knows how to manage files. Why build anything more complex?

Because files solve only one problem — persistence. They store data across power cycles. Everything else you need when building real software, files give you nothing.

Why do files break under concurrent access?

Imagine two users simultaneously updating their profile on your application. Both read the file, modify their section, and write it back. The second write overwrites the first. One user's changes are silently lost.

This is called a race condition, and it's fundamental to how files work. A file has no concept of "who else is currently reading or writing." Multiple processes can open the same file simultaneously and clobber each other without any warning.

You could add locking — before writing, acquire a lock on the file; release it when done. But now every write to any part of the file blocks all reads and all other writes. You've serialized access to your entire dataset. One slow write brings everything else to a halt.

Databases solve this at a fine-grained level: lock only the rows being modified, not the whole dataset. Two users updating different rows proceed in parallel. Two users updating the same row are safely serialized, with the second waiting for the first to finish.

Why do files give you no way to find data without reading all of it?

A file is a sequence of bytes. To find a user with id = 5000, you must read through the entire file, byte by byte, until you find the record you want. This is an O(n) scan — it gets slower as the file grows, linearly.

A file with one million user records requires reading all one million records to find one. A file with one billion records is a thousand times slower.

Databases solve this with indexes — data structures that let you jump directly to the record you want in O(log n) time. Find user 5000 out of one billion in ~30 comparisons instead of ~500 million reads.

Why does a file not protect you from partial writes?

Suppose your application is updating a bank account — debiting one account and crediting another. The process writes the debit to the file and then crashes before writing the credit. The money is gone from one account and hasn't appeared in the other. The data is now inconsistent.

With a file, there is no way to say "these two writes must either both happen or neither happen." There is no concept of grouping writes into an atomic unit.

Databases solve this with transactions. You wrap both writes in a transaction. If anything goes wrong before the transaction commits, the database rolls back the partial work. The data is always left in a consistent state.

Why does a file have no schema?

A file is bytes. It has no opinion about what those bytes mean, whether they're valid, or what relationships they have with bytes in other files. Nothing prevents you from writing "age": -500 or leaving a required field empty or storing a reference to a user that doesn't exist.

Over time, without enforcement, data becomes garbage. Applications diverge in what they write. Old code assumes one format; new code assumes another. The file accumulates contradictions.

Databases enforce a schema — a contract about what data is valid. Constraints (NOT NULL, UNIQUE, FOREIGN KEY) are enforced at write time. You cannot store an order for a customer that doesn't exist. You cannot store a null in a column that requires a value. The database is the last line of defense against corrupt data.

Why does a file have no query language?

To answer "give me all users who signed up in the last 30 days and have made at least one purchase," you'd have to write custom code: read all users, filter by date, cross-reference the purchases file, deduplicate. Every question about your data requires custom programming.

SQL lets you express that question in one readable statement. The database figures out the most efficient way to execute it — which indexes to use, which tables to join and in what order, whether to sort or hash. You state what you want; the query planner figures out how to get it.

The mental model

Files give you:    persistence only
Databases give you:
  ├── Concurrent access → row-level locking, not file-level
  ├── Fast lookup → indexes, O(log n) instead of O(n) scan
  ├── Atomic writes → transactions, all-or-nothing
  ├── Data integrity → schemas, constraints, foreign keys
  └── Declarative queries → SQL, query planner handles execution

A database is a file system with answers to every problem files don't solve.

Databases exist because the gap between "store bytes" and "safely manage structured data at scale with concurrent access" is enormous — and every application would otherwise have to bridge that gap itself, badly, from scratch.