AttemptIQ: Systems Thinking From First Principles

A reverse-engineering of every decision in AttemptIQ, down to its atomic justification.

Preface

This book does not explain what AttemptIQ does. It explains why every decision in AttemptIQ exists — starting from the problem, not the solution.

The reader is assumed to be intelligent but new to production systems thinking. After reading this book, you should be able to sit in front of a blank whiteboard, hear a product requirement, and reason your way to a production-grade architecture — on your own.

The method is simple: start from the problem. Every time you see a technical decision, ask why until there is nothing left to ask. The insight lives at the bottom of that chain.

AttemptIQ is a UPSC CSE + CSAT exam prep platform. Students practice questions, take mock tests, and get analytics on patterns they cannot see themselves: how often they get lucky on guesses, how pressure affects their accuracy, which trap types keep fooling them. It runs on a single Hetzner VPS. It handles thousands of users. It was built by one engineer. Every trade-off in this book is a real trade-off made under those constraints.

Chapter 1: Foundation — The System as a Whole

What Is AttemptIQ, Exactly?

Before reasoning about any part, you need to see the whole.

AttemptIQ is a monolith. One codebase. One deployment. One process manager. It does not split into microservices. It does not use managed cloud services for its database. It runs on a bare Linux server in a Hetzner data center.

Here is every moving part, mapped:

Internet
  │
  ▼
Cloudflare (DNS + DDoS shield + edge cache)
  │
  ▼
Nginx on VPS (SSL termination + rate limiting + reverse proxy)
  │
  ▼
Next.js 14 App (4 PM2 cluster processes, port 3000)
  ├── Server Components (SSR — rendered on server, HTML sent to browser)
  ├── API Routes (thin: validate → service → return)
  ├── Middleware (cookie check → redirect or pass)
  │
  ├── Services (all business logic, dependency-injected repos)
  │     ├── sessions.service.ts
  │     ├── attempts.service.ts
  │     ├── payments.service.ts
  │     ├── analytics.service.ts
  │     └── ...
  │
  ├── Repositories (all DB queries, injectable, mockable)
  │     ├── sessions.repo.ts
  │     ├── questions.repo.ts
  │     ├── analytics.repo.ts
  │     └── ...
  │
  └── lib/
        ├── db/client.ts    → pg Pool → pgBouncer (port 6432)
        ├── redis/client.ts → ioredis → Redis (port 6379)
        └── auth/index.ts   → Better Auth (sessions, OAuth)

pgBouncer (port 6432, transaction pooling)
  │
  ▼
PostgreSQL 16 (port 5432, localhost only)
  ├── 24 application tables
  └── 6 materialized views (analytics engine)

Redis (port 6379, localhost only)
  ├── BullMQ job queues (7 queues)
  └── Application cache (leaderboard, analytics)

BullMQ Worker (separate PM2 process: attemptiq-worker)
  ├── analytics-refresh → REFRESH MATERIALIZED VIEW × 6
  ├── email → transactional email via Brevo SMTP
  ├── revision-list → daily revision cache generation
  ├── payment-recon → Razorpay reconciliation
  └── db-backup → pg_dump to Cloudflare R2

Cloudflare R2 (S3-compatible object storage)
  └── PDF files, pg_dump backups

Ingestion Pipeline (Python, separate process, admin-triggered)
  └── PDF → Gemini AI extraction → Claude enrichment → DB staging → admin review → approved questions

Two SaaS products share this single VPS. AttemptIQ runs on port 3000. ScoreResume runs on port 3001. They share the same Nginx process but have strictly separate PM2 processes, databases, Redis namespaces, and Nginx configs. The operational rule is absolute: never touch ScoreResume when working on AttemptIQ. This is not a preference — it is documented as a non-negotiable constraint because the consequence of contamination (dropping the wrong database, reloading the wrong PM2 process) is an immediate production outage of a live product.

The Request Lifecycle

Let us trace a single request — a student submitting an answer to a question — from the browser keystroke to the database write and back.

Step 1: The browser. The student selects option "B" and clicks "Submit." The JavaScript event handler fires. The client sends:

POST /api/attempts/submit
Content-Type: application/json
Cookie: better-auth.session_token=<encrypted-token>

{
  "questionId": "uuid",
  "sessionId": "uuid",
  "selectedAnswer": "B",
  "solveMethod": "elimination",
  "optionsEliminated": ["A", "C"],
  "timeTakenSeconds": 45
}

Step 2: Cloudflare. The request arrives at Cloudflare's edge. Cloudflare checks if it is a bot attack. It is not. The request passes through. Cloudflare forwards it to the VPS with the real client IP in the CF-Connecting-IP header.

Step 3: Nginx. Nginx receives the request on port 443. It terminates TLS (decrypts the HTTPS). It checks the rate limit zone. /api/attempts/submit allows 120 requests per hour per user. The student has not exceeded this. Nginx proxies the request to localhost:3000.

Step 4: Next.js Middleware. Before the API route handler runs, middleware.ts intercepts the request. It reads the session cookie using getSessionCookie(request) — a fast, edge-compatible function that checks if the cookie exists. This is not a database lookup. It is a cookie presence check. If no cookie exists, the user is redirected to /login. The cookie exists. Middleware passes the request through.

Step 5: The API Route Handler. The request reaches app/api/attempts/submit/route.ts. Three things happen in strict order:

// 1. Auth — full DB-backed session validation
const session = await auth.api.getSession({ headers: await headers() });
if (!session) return Response.json({ error: 'Auth required' }, { status: 401 });

// 2. Validate — Zod schema, no DB call
const parsed = submitSchema.safeParse(body);
if (!parsed.success) return Response.json({ error: 'Invalid request' }, { status: 400 });

// 3. Call service
const result = await attemptsService.submitAttempt(session.user.id, parsed.data);

Step 6: The Service. attemptsService.submitAttempt() executes the business logic: validate session ownership, check time limit against the server clock, check if this question was already answered, fetch the correct answer from the database server-side, compute correctness, persist the attempt.

Step 7: The Repository. The service calls attemptsRepo.insert(). The repository executes a parameterized INSERT query against the database.

Step 8: pgBouncer. The INSERT travels from the Node.js pg Pool connection to pgBouncer on port 6432. pgBouncer is a connection pool proxy. It holds a small number of real PostgreSQL connections and multiplexes many application connections through them. The query is forwarded to PostgreSQL.

Step 9: PostgreSQL. The INSERT executes. The row is written to user_attempts. PostgreSQL returns success.

Step 10: The response. The service returns an AttemptResult object containing isCorrect, correctAnswer, explanation, and trapTypes. The route handler wraps it in { data: result } and returns HTTP 200. The response flows back through pgBouncer → Next.js → Nginx → Cloudflare → browser. The browser renders the feedback: correct or wrong, with the explanation.

Total elapsed time: typically 80–200ms.

The mental model to take from this: Every request passes through at least six distinct systems before it reaches the database and returns. Each system has one job. Cloudflare absorbs attacks. Nginx terminates SSL and rate-limits. Middleware does fast cookie checks. The route handler validates input. The service runs business logic. The repository stores data. The layering is not ceremony — each layer catches a different class of failure before it can propagate deeper.

Chapter 2: Identity and Trust

Authentication — What It Means to Prove Who You Are to a Machine

The Problem First

Imagine you call a bank. The bank has no idea who is on the phone. You say "I am Ramesh Kumar." The bank needs to verify this. They ask for your account number, your date of birth, your mother's maiden name. This is authentication: proving you are who you claim to be.

Now imagine the bank has to do this verification for every single sentence you utter in the conversation. That would be insane. So instead, after verification, they put you in a "verified state" for the duration of the call. You authenticated once; you stay authenticated for that session.

HTTP breaks this model entirely. HTTP is stateless. Every request is born knowing nothing about the previous request. There is no concept of a "session" in HTTP itself. Each request is a fresh phone call with no memory of the last one.

This is the root cause of all authentication complexity in web systems.

The Naive Solution (and why it fails)

What would a junior developer do? Send the username and password with every request.

POST /api/attempts/submit
Authorization: Basic cmFtZXNoOm15cGFzc3dvcmQ=

This base64-encodes "ramesh:mypassword" and sends it with every request. It fails for multiple reasons:

• The password now travels over the network with every single API call. Every call is an opportunity to intercept it.
• The server must verify the password against the database on every request — this is expensive (bcrypt hashing takes ~100ms by design, to slow down brute-force).
• There is no way to "log out" — you would have to change the password.
• There is no way to expire a session — once you have the password, you have it forever.

The Insight

The solution is to separate authentication from authorization. Authenticate once, get a proof token. Present the proof token on subsequent requests. The server can verify the proof token cheaply, without database lookups for every request.

This proof token is a session token. It works like a wristband at a concert. Security checks your ID once at the gate (authentication). They put a wristband on you. For the rest of the night, staff just checks the wristband (authorization) — no ID needed.

The wristband must be:

• Hard to forge (cryptographically signed or random enough to be unguessable)
• Tied to an expiry (concerts end)
• Revocable (security can cut it off)

The Solution AttemptIQ Uses

AttemptIQ uses Better Auth — a Next.js-native authentication library. On successful login, Better Auth creates a session token and stores it in a cookie:

// lib/auth/index.ts
export const auth = betterAuth({
  advanced: {
    defaultCookieAttributes: {
      sameSite: 'lax' as const,
      httpOnly: true,
    },
  },
  session: {
    modelName: 'ba_sessions',
  },
});

Two cookie attributes matter enormously:

httpOnly: true — This cookie cannot be read by JavaScript. If an attacker injects malicious JavaScript into the page (XSS), the JavaScript cannot steal the session cookie. The browser will send it automatically, but JavaScript cannot access it via document.cookie.

sameSite: 'lax' — This cookie is not sent on cross-site requests initiated by third-party sites (like clicking a link from another domain to perform a state-changing action). This blocks Cross-Site Request Forgery (CSRF) attacks.

The session token is stored in the ba_sessions table in PostgreSQL. On every authenticated API request, the server reads the cookie and validates it against the database:

const session = await auth.api.getSession({ headers: await headers() });

This is a database lookup on every request. That is the correct trade-off — the cost of a single indexed lookup (< 5ms) for the security of knowing the session is valid and has not been revoked.

Concurrent session limiting:

What if Ramesh logs in from two browsers and then loses his phone? The attacker now has a valid session. The limit of 2 concurrent sessions means a new login automatically revokes the oldest session:

// lib/auth/index.ts — before hook
if (count >= 2) {
  await db.query(
    `DELETE FROM ba_sessions WHERE id = (
      SELECT id FROM ba_sessions WHERE "userId" = $1
      ORDER BY "createdAt" ASC LIMIT 1
    )`,
    [user.id]
  );
}

If you log in from a third device, your oldest session (potentially the stolen one) is gone.

The Tradeoffs

This approach requires a database lookup on every request. At scale, this becomes a concern. The mitigation is that the session table is tiny and indexed on token — lookups take microseconds, not milliseconds, once the row is cached in PostgreSQL's buffer pool.

The alternative — stateless JWT tokens — eliminates the database lookup but makes session revocation impossible without additional infrastructure. If a user's JWT is stolen, you cannot revoke it until it expires. For an exam prep app where a student's account and subscription could be hijacked, this is an unacceptable trade-off.

The Mental Model (take this with you)

HTTP is stateless. Sessions are a lie we maintain with cookies and database lookups. The security of a session depends entirely on: (1) the randomness of the token, (2) the protection of the cookie from theft (httpOnly, sameSite), and (3) the ability to revoke it quickly. Every authentication system is a balance between these three properties and the cost of maintaining them.

If this were 10x scale

At 10x users, the session lookup becomes the hot path. Redis session caching would reduce the database hit to a memory lookup. The trade-off: you now need to invalidate the Redis cache when a session is revoked, or accept a brief window where a revoked session still works (up to the cache TTL).

Authorization — The Difference Between Identity and Permission

Authentication answers "who are you?" Authorization answers "what are you allowed to do?"

These are separate problems. A user can be fully authenticated — the server knows exactly who they are — and still be forbidden from an action. They are paying a free-tier user. They want to access analytics. Analytics is a paid feature.

AttemptIQ has four distinct authorization levels:

1. Public — No auth required. /upsc/[subject]/[year]/[slug] pages. Google can crawl these.
2. Authenticated — Must have a valid session. /dashboard, /practice, /mock-test.
3. Paid — Must have a valid session AND plan_tier != 'free'. Analytics, unlimited mock tests.
4. Admin — Must have a valid session AND is_admin = true in ba_users. All /admin/* routes.

The critical rule is: the middleware fast-checks only the session cookie. The actual authorization check must happen in the route handler or layout, not just the middleware.

Why? Middleware runs on Next.js's Edge Runtime. It cannot make database calls. It can only inspect cookies and headers. So middleware says "you have a cookie, you may proceed." The route handler says "you have a cookie AND you are admin AND your session is valid in the database AND your plan is paid."

This is why requireAdmin() exists as a separate function:

// lib/auth/admin.ts
export async function requireAdmin(): Promise<{ userId: string } | null> {
  const session = await auth.api.getSession({ headers: await headers() });
  if (!session) return null;

  const { rows } = await db.query<{ is_admin: boolean }>(
    'SELECT is_admin FROM ba_users WHERE id = $1',
    [session.user.id],
  );

  if (!rows[0]?.is_admin) return null;
  return { userId: session.user.id };
}

This function does not trust the session claims alone. It goes to the database and confirms is_admin. Why? Because the session object returned by Better Auth reflects what was true when the session was created. If an admin is demoted after the session was created, the session might still show them as admin. The database check reflects current reality.

The blocked user problem:

What about users who are blocked? The middleware cannot check is_blocked because that requires a database lookup. So the block check lives in the authenticated app layout:

// app/(app)/layout.tsx
// DB query checks is_blocked, plan_tier, is_admin, onboarding_completed
// If is_blocked → redirect('/blocked')

This is correct. A blocked user still has a valid session cookie. The middleware passes them through. The layout's DB query catches them before any page renders.

The mass-assignment attack:

One of the most common authorization failures is mass-assignment. An attacker sends:

{ "username": "ramesh", "is_admin": true, "plan_tier": "annual" }

If the server spreads this into a database update without whitelisting fields, the attacker upgrades themselves to admin and paid status for free. AttemptIQ prevents this with Zod schemas — every user-editable API route has an explicit schema that does not include is_admin, plan_tier, or plan_expires_at:

// CORRECT — only whitelisted fields can be updated
const schema = z.object({
  username: z.string().min(3).max(30).optional(),
  is_public: z.boolean().optional(),
  // is_admin, plan_tier, plan_expires_at are NOT here
});

Session Management — The Two-Layer Cookie Model

AttemptIQ uses two distinct cookie systems that must not be confused:

1. ba_sessions — Better Auth's session table. Stores the actual auth session. Queried on every authenticated request.
2. aiq_ob — A simple first-party cookie set after onboarding completion. Value: "1". The middleware reads this to redirect new users to onboarding.

The aiq_ob cookie illustrates an important pattern: using cookies as a fast signal for middleware routing, while the actual source of truth remains in the database (user_profiles.onboarding_completed). The cookie can lie — but the worst case is an extra redirect, not a security breach.

There is a documented gotcha here. After the onboarding form submits, the API sets aiq_ob=1 in the response. The next navigation must use window.location.href, not router.push(). Why? Because Next.js App Router client-side navigation can race against Set-Cookie headers — the RSC request might fly before the browser processes the new cookie, and middleware sees the old state. A full page navigation (window.location.href) forces the browser to re-evaluate all cookies from scratch.

This is the kind of bug that appears rarely, in specific browser-timing conditions, and is invisible in local testing. It was caught and documented. The fix is four characters: using href instead of push.

Chapter 3: The Test Delivery Pipeline

How a Test Session Begins, Lives, and Dies

The Problem First

A student wants to take a practice session. They select "History," 20 questions, medium difficulty. They click "Start." What happens in the system?

The naive answer: "Look up 20 history questions and show them one at a time." But this naive answer breaks immediately in production. What if the student takes the same subject twice? They should not see the same questions. What if two students start sessions simultaneously — should they get the same questions? What if a student closes the tab mid-session and reopens? The session should resume exactly where they left off, with the same questions.

A session is not just a list of questions. It is a stateful object with a lifecycle: created → in-progress → submitted.

The Insight

The key insight is that question selection must happen at session creation time, not at each question fetch. The question IDs are locked in at the moment the session is created, stored in the database, and served sequentially from there. The session is the contract: these are your 20 questions, in this order, and they will not change.

This eliminates an entire class of problems:

• No duplicate questions across refreshes
• Resumable sessions (reload the page, pick up where you left off)
• Server can verify "has this question already been answered in this session?" against a fixed list

The TOCTOU Race Condition

Here is the most dangerous bug in the entire session creation flow. It lives at the intersection of database reads and writes.

Imagine this scenario: A free user is allowed 2 lifetime practice sessions. They have used 1. They open two browser tabs and click "Start" in both simultaneously. Two concurrent requests hit the server:

Request A: SELECT COUNT() FROM sessions WHERE user_id = $1 → returns 1 (under limit) Request B: SELECT COUNT() FROM sessions WHERE user_id = $1 → returns 1 (under limit) Request A: INSERT INTO sessions → succeeds (count is now 2) Request B: INSERT INTO sessions → succeeds (count is still 1 in B's view)

Both requests saw count < limit and both inserted. The user now has 3 sessions when the limit is 2. This is a TOCTOU (Time Of Check To Use) race condition — the check and the use are not atomic.

The junior fix is to add a database lock. But database locks are expensive and complicate things enormously.

The Solution AttemptIQ Uses

The elegant fix is to embed the count check inside the INSERT itself, making the entire operation atomic:

-- sessions.repo.ts — createPracticeSession
WITH selected AS (
  SELECT ARRAY_AGG(q.id) AS qids
  FROM (
    SELECT q.id FROM questions q
    WHERE q.subject = $2
      AND q.is_approved = true
    ORDER BY RANDOM()
    LIMIT $3
  ) q
)
INSERT INTO sessions (user_id, session_type, question_ids)
SELECT $1, 'practice', s.qids
FROM selected s
WHERE s.qids IS NOT NULL
  AND array_length(s.qids, 1) > 0
  AND (SELECT COUNT(*) FROM sessions WHERE user_id = $1 AND session_type = 'practice') < 2
RETURNING id;

The WHERE clause evaluates atomically during the INSERT. PostgreSQL holds a row-level lock while executing this. If two requests race, one wins and the other sees 0 rows returned — which the service interprets as "limit reached":

// sessions.service.ts
const sessionId = await this.sessionsRepo.createPracticeSession({ ... });
if (!sessionId) {
  throw new SessionLimitReachedError();
}

But there is still a subtle problem. PostgreSQL's serialization of the INSERT is not guaranteed to fully prevent TOCTOU across concurrent transactions that each check the count independently. For the limit guard to be truly atomic, AttemptIQ adds a per-user advisory lock inside a transaction:

// sessions.repo.ts — _lockedInsertSession
private async _lockedInsertSession(userId: string, sql: string, params: unknown[]): Promise<string | null> {
  const client = await db.connect();
  try {
    await client.query('BEGIN');
    await client.query('SELECT pg_advisory_xact_lock(hashtext($1::text)::bigint)', [userId]);
    const { rows } = await client.query<{ id: string }>(sql, params);
    await client.query('COMMIT');
    return rows[0]?.id ?? null;
  } catch (err) {
    await client.query('ROLLBACK');
    throw err;
  } finally {
    client.release();
  }
}

pg_advisory_xact_lock acquires a lock keyed to the hash of the user's ID. Only one transaction per user can hold this lock at a time. Any concurrent request for the same user blocks until the first transaction commits. The lock is automatically released on COMMIT or ROLLBACK — making it safe with pgBouncer's transaction pooling mode.

This is defense in depth: the embedded COUNT guard in the INSERT is the first line of defense. The advisory lock is the second. Even if the COUNT guard had a race window, the lock prevents two transactions from reaching it simultaneously.

State Management — Where Does the Live Test Live?

A mock test is in progress. The student is on question 37 of 100. They have been going for 45 minutes. The time limit is 2 hours. Where does this state live?

The naive answer: in memory on the server. This breaks the moment the server process restarts, or when PM2 runs 4 cluster instances and the student's next request hits a different instance.

The correct answer: in the database. The session row in PostgreSQL is the authoritative state. It contains:

• question_ids: the fixed list of questions
• started_at: the server-side timestamp when the session began
• is_submitted: whether the session has been submitted
• tab_switch_count: how many times the student switched tabs

The client holds only display state: which question they are currently viewing, the countdown timer. The timer is derived from started_at — the client computes time_limit_seconds - elapsed_seconds and displays it. But the server recomputes elapsed time from started_at on every answer submission:

// attempts.service.ts
if (session.time_limit_seconds) {
  const elapsedMs = Date.now() - new Date(session.started_at).getTime();
  const elapsedSeconds = Math.floor(elapsedMs / 1000);
  // +10s grace for network lag (kept tight to prevent cheating)
  if (elapsedSeconds > session.time_limit_seconds + 10) {
    throw new SessionExpiredError();
  }
}

The client timer is cosmetic. The server timer is authoritative. A student cannot cheat by pausing the JavaScript timer, putting the browser in the background, or manipulating the client-side state. The server uses its own clock.

Idempotency — What Happens If Submit Is Clicked Twice?

The "Submit" button is clicked. The request is in flight. The student's internet blips. The request might have succeeded. They click Submit again. Now two requests race to submit the session.

The solution is that submitSession is an idempotent operation:

// sessions.repo.ts
async submitSession(id: string, userId: string): Promise<boolean> {
  const { rowCount } = await db.query(
    `UPDATE sessions
     SET is_submitted = true, ended_at = now()
     WHERE id = $1 AND user_id = $2 AND is_submitted = false`,
    [id, userId],
  );
  return (rowCount ?? 0) > 0;
}

The WHERE clause includes is_submitted = false. The first submit changes the row. The second submit hits a row where is_submitted = true — the WHERE clause matches nothing, rowCount = 0, and the service throws SessionAlreadySubmittedError. The API route catches this and returns 409 Conflict. The client knows: session is already submitted, show the results page.

Answer locking — the same problem for individual questions:

// attempts.service.ts
const alreadyAnswered = await this.attemptsRepo.existsForQuestion(dto.sessionId, dto.questionId);
if (alreadyAnswered) throw new AnswerAlreadyLockedError();

And at the database level, user_attempts has a unique constraint on (user_id, question_id, session_id). Even if the service check races, the database INSERT will fail with a uniqueness violation. The service catches this and returns the appropriate error.

The correct_answer security invariant:

This is Rule 1 in AttemptIQ, non-negotiable: correct_answer is never sent to the client before the answer is locked.

The flow is:

1. Client sends { questionId, selectedAnswer } — no correct answer
2. Server fetches correct_answer from DB server-side
3. Server computes is_correct = selectedAnswer === correct_answer
4. Server persists the attempt
5. Server returns { isCorrect, correctAnswer, explanation } — only NOW

If correct_answer were sent to the client with the question, any student who inspected the network tab would see the answer before guessing. This would make the platform useless as an assessment tool. The invariant is enforced by never including correct_answer in the QuestionPublic type — the type that APIs return for active sessions.

The Mental Model (take this with you)

A session is a contract, not a stream. Question selection happens once, upfront, and is immutable. State lives in the database, not in application memory. Every state-changing operation must be idempotent — clicked once or clicked ten times, the outcome must be the same. The server clock, not the client clock, determines what is allowed.

If this were 10x scale

The ORDER BY RANDOM() query to select questions scales poorly. At 100,000 questions, ORDER BY RANDOM() must generate a random value for every row and sort them. This becomes a full-table scan. The fix is to use TABLESAMPLE or a pre-computed random ordering, or to randomly select question IDs from a set of known IDs using application-level sampling rather than database-level sorting.

Chapter 4: Content Ingestion

How Questions Enter the System

The Problem First

AttemptIQ needs thousands of UPSC questions. These questions come from scanned PDFs of previous-year question papers. Raw PDFs contain text that is formatted for human readers: numbered questions, options labeled A to D, answer keys at the back. No structure a database can directly consume.

Someone needs to transform unstructured PDF text into structured database rows. This is the ingestion pipeline.

The Naive Solution (and why it fails)

A junior developer might do this synchronously, inline with an HTTP request. Admin clicks "Upload PDF" → server processes it → done. This fails immediately for large PDFs:

• A 200-question PDF takes minutes to process with AI enrichment. HTTP requests time out after 30–60 seconds.
• If the server crashes mid-processing, all work is lost.
• The admin user's browser is frozen waiting.
• There is no visibility into progress.

The Insight

Slow, fallible, expensive operations must be decoupled from the request that triggers them. The trigger (PDF upload) should be fast and synchronous. The processing (AI extraction, enrichment, DB insertion) should be asynchronous, retryable, and observable.

This is the producer-consumer pattern, also known as the queue-based decoupling pattern.

The Solution AttemptIQ Uses

The ingestion pipeline is a Python process (ingestion/) that runs independently from the Next.js app. It is not a BullMQ worker — it is a command-line script that can be triggered by the admin API or run manually. Here is the flow:

Admin uploads PDF
  → POST /api/admin/ingestion/upload
    → stores PDF to Cloudflare R2
    → creates ingestion_jobs row (status: 'queued')
    → spawns python ingest.py --pdf <r2-url> --job-id <uuid>

ingestion/ingest.py:
  1. Load PDF (or fixture data for testing)
  2. Extract raw question text using Gemini AI (or regex parsers)
  3. For each raw question:
     a. Enrich using Claude API (classify subject, topic, difficulty, trap_types, etc.)
     b. Validate enriched data
     c. Insert into questions table (is_approved = false)
  4. Update ingestion_jobs.status to 'complete'

The questions land in the database with is_approved = false. They are visible in the admin staging area but not served to students. An admin reviews them, edits if needed, and approves them.

Why does it commit in batches?

The ingestion script commits every 50 successful inserts and then reconnects to the database:

# ingest.py
if inserted % args.batch_size == 0:
    conn.commit()
    conn.close()
    conn = psycopg2.connect(database_url)

Why reconnect? Long-running connections can exceed pgBouncer's server_lifetime — the maximum time pgBouncer keeps a server connection alive. If the connection dies mid-run, all uncommitted work is lost. By committing and reconnecting every 50 questions, the worst-case data loss is 50 questions, and the reconnection resets the timer.

Retry logic — why failures are inevitable:

The Claude API call in enrichment can fail: rate limits, transient API errors, network blips. The retry logic is exponential backoff:

# ingestion/enrich.py (pattern from docs/resilience.md)
def call_claude_with_retry(prompt, max_retries=3):
    for attempt in range(max_retries):
        try:
            return client.messages.create(...)
        except anthropic.RateLimitError:
            wait = 2 ** attempt * 10  # 10s, 20s, 40s
            time.sleep(wait)
        except anthropic.APIError as e:
            if attempt == max_retries - 1:
                raise
            time.sleep(5)
    raise Exception("Claude API failed after retries")

Why exponential backoff? When an API rate-limits you, it means it is overloaded. If every client retries immediately, they all hit the API at the same time, making the overload worse. Exponential backoff spreads retries out: first retry at 10s, second at 20s, third at 40s. The load on the API decreases as time passes, giving it time to recover.

Failed questions are not lost:

Questions that fail enrichment are written to a failed_YYYY-MM-DD.jsonl file. The --replay-failed flag allows reprocessing them later:

python ingest.py --replay-failed logs/failed_2026-04-28.jsonl --exam-id <uuid> --paper GS1

The replay loader filters out unrecoverable fragments (raw text under 20 characters, options missing) before feeding them back through enrichment. This is the difference between a pipeline that loses data on failure and one that degrades gracefully.

The search vector:

Every question has a search_vector column of type tsvector. This is PostgreSQL's full-text search data structure. It is auto-populated by a database trigger on every insert or update to the questions table:

-- trigger: questions_search_update BEFORE INSERT OR UPDATE
-- auto-updates search_vector

The application code never sets search_vector manually. It queries it:

-- questions.repo.ts
SELECT * FROM questions WHERE search_vector @@ plainto_tsquery($1)

A GIN index (idx_questions_search) on search_vector makes this fast — it is an inverted index, the same data structure that powers Google's search. For 3,000–10,000 questions, this is more than sufficient.

The Tradeoffs

The AI enrichment costs money. The docs note this is a one-time cost of approximately ₹400 for the entire initial dataset. After that, questions are human-reviewed before approval. The cost is bounded because enrichment runs once per question, not continuously.

The pipeline is currently admin-triggered, not automated. There is no automated ingestion of new papers. This is a deliberate choice — every question goes through human review. Automated ingestion of unreviewed questions would risk bad data reaching students.

The Mental Model (take this with you)

Slow operations belong in queues, not in request handlers. The ingestion pipeline is a producer-consumer system: the admin is the producer (triggers the job), the Python worker is the consumer (does the work). Decoupling them with a job record in the database means the work survives server restarts, can be monitored, and can be retried without human intervention.

Chapter 5: The Data Layer

Schema Design — Why the Tables Are Shaped the Way They Are

The questions table:

questions (
  id uuid PRIMARY KEY,
  body text NOT NULL,
  options jsonb NOT NULL,  -- { "A": "...", "B": "...", "C": "...", "D": "..." }
  correct_answer text NOT NULL,
  subject text NOT NULL,
  topic text,
  difficulty text CHECK IN ('easy', 'medium', 'hard'),
  trap_types text[],
  is_approved boolean DEFAULT false,
  search_vector tsvector,  -- auto-populated by trigger
  slug text NOT NULL UNIQUE,
  ...
)

Why is options stored as JSONB rather than four separate columns (option_a, option_b, option_c, option_d)? Because the options are always read and written together, they have no independent query patterns, and they need to be sent to the client as a single object. JSONB stores them together, retrieves them in one operation, and serializes trivially to the JSON the client expects.

Why is correct_answer a plain text column ('A', 'B', 'C', 'D') rather than a foreign key to some options table? Because the answer is simple string equality — selectedAnswer === question.correct_answer. Adding a foreign key would add a join with no benefit.

The sessions table:

sessions (
  id uuid PRIMARY KEY,
  user_id text NOT NULL,
  session_type text CHECK IN ('practice', 'mock_test'),
  question_ids uuid[],   -- the fixed list of questions, pre-selected
  started_at timestamptz DEFAULT now(),
  ended_at timestamptz,
  is_submitted boolean DEFAULT false,
  time_limit_seconds integer,   -- NULL = untimed
  tab_switch_count integer DEFAULT 0,
  ...
)

Notice what is NOT in this table: subject, question_count, status. These are common traps for beginners.

subject is not stored because a session can span multiple subjects (mock tests do). The subject comes from the questions themselves.

question_count is not stored because array_length(question_ids, 1) computes it. Storing it would create an invariant to maintain — what happens if question_ids is updated but question_count is not?

status is not stored as a separate column because is_submitted is sufficient. A session is either in-progress (is_submitted = false) or done (is_submitted = true). Adding a status enum adds complexity without adding information.

The user_attempts table — denormalized by design:

user_attempts (
  id uuid PRIMARY KEY,
  user_id text NOT NULL,
  question_id uuid NOT NULL,
  session_id uuid NOT NULL,
  selected_answer text,    -- NULL = skipped
  is_correct boolean,
  solve_method text,
  time_taken_seconds integer,
  attempted_at timestamptz DEFAULT now(),
  session_type text CHECK IN ('practice', 'mock_test'),  -- DENORMALIZED
  UNIQUE (user_id, question_id, session_id)
)

session_type appears in both sessions and user_attempts. This is intentional denormalization. The analytics engine (materialized views) needs to distinguish practice attempts from mock test attempts. If session_type lived only in sessions, every analytics query would need to join user_attempts to sessions. At scale, with hundreds of thousands of attempts, that join is expensive.

By denormalizing session_type into user_attempts, the materialized views can aggregate directly on user_attempts without any join. The cost is a small redundancy — the session_type must be copied at insertion time.

The N+1 Problem

The N+1 problem is one of the most common performance bugs in web applications. Here is how it manifests:

You want to show a session result with all 20 questions and their answers. A naive implementation does this:

const session = await sessionsRepo.findById(sessionId);
const attempts = await attemptsRepo.listBySession(sessionId);

// N+1: one query per question
for (const attempt of attempts) {
  const question = await questionsRepo.findById(attempt.question_id);
  // ...
}

This executes 1 query for the session, 1 query for attempts, and then N queries for N questions — 22 queries total for a 20-question session. At 100 concurrent users viewing results pages, this is 2,200 database queries for something that should take 3.

AttemptIQ avoids this by fetching all question data in bulk using findAllByIds:

// sessions.service.ts — getFullSessionResult
const [attempts, publicQuestions, questionsWithAnswers] = await Promise.all([
  this.attemptsRepo.listBySession(sessionId),
  this.questionsRepo.findAllByIds(questionIds),
  this.questionsRepo.findAllByIdsWithAnswers(questionIds),
]);

// Build maps for O(1) lookup
const attemptMap = new Map(attempts.map(a => [a.question_id, a]));
const publicMap  = new Map(publicQuestions.map(q => [q.id, q]));
const answerMap  = new Map(questionsWithAnswers.map(q => [q.id, q]));

Three queries, regardless of session size. The Promise.all fires all three concurrently. And the Map lookups are O(1) — no loops with database calls.

Indexes — What Makes Queries Fast at the Hardware Level

An index is a sorted data structure maintained alongside a table. Without an index, finding a row requires reading every row in the table (sequential scan). With an index, finding a row is binary search over sorted data — O(log n) instead of O(n).

But why does sorting help? At the hardware level, reading sequential disk pages is fast because the disk head does not need to move. Random reads are slow because the disk head must seek. An index organizes data so that queries read sequential blocks, minimizing seeks.

AttemptIQ's most important indexes:

-- Sessions: finding active sessions for a user
-- Used every time a student continues a session
idx_sessions_active: (user_id, is_submitted) WHERE is_submitted = false

-- Attempts: analytics queries
-- Used by every materialized view refresh
idx_attempts_user_q: (user_id, question_id)
idx_attempts_user_time: (user_id, attempted_at)

-- Questions: full-text search
-- GIN index — inverted index for text search
idx_questions_search: GIN (search_vector)

-- Questions: fetching only approved questions
idx_questions_approved: (is_approved)

The idx_sessions_active is a partial index: it only includes rows where is_submitted = false. Submitted sessions are never looked up in the hot path. A partial index is smaller, faster to scan, and cheaper to maintain.

Transactions — When You Need All-or-Nothing

The webhook payment handler illustrates why transactions exist. Consider what happens without a transaction:

1. UPDATE payments SET status = 'captured' WHERE id = $1 AND status = 'pending'
   → succeeds

2. UPDATE user_profiles SET plan_tier = 'paid' WHERE user_id = $1
   → server crashes here

The payment is marked captured. The user's plan is not upgraded. The user paid but cannot access paid features. The payment record says captured. Your support team is confused.

With a transaction:

// payments.service.ts — handleWebhook
await client.query('BEGIN');

// All three updates or none
await client.query(`UPDATE payments SET status = 'captured' ...`);
await client.query(`UPDATE user_profiles SET plan_tier = ... `);
await client.query(`UPDATE webhook_events SET processed = true ...`);

await client.query('COMMIT');

If the server crashes after the second UPDATE but before COMMIT, PostgreSQL rolls back the entire transaction when it restarts. The user's payment row goes back to pending. Dodo retries the webhook. The transaction runs again. This time it succeeds. The user gets their plan.

COMMIT is atomic — it either applies all changes or none. There is no "half-committed" state.

Materialized Views — The Analytics Engine

AttemptIQ's analytics are complex: accuracy per subject, trap susceptibility by type, performance under pressure vs practice, solve method evolution over time. Computing these on-the-fly at request time would require expensive aggregation queries over hundreds of thousands of rows.

The solution is materialized views (MVs): pre-computed query results stored as regular tables. An MV is re-computed on schedule, not on demand.

-- mv_user_subject_stats: accuracy per user per subject
-- Refreshed every 2 hours
-- Powers: Exam Readiness Score, subject leaderboard

Reading from an MV is as fast as reading from a regular table. The aggregation work happened during the refresh. A query that would take 500ms against raw data takes 5ms against an MV.

The tradeoff: staleness. The MV is always up to 2 hours behind real-time. For exam analytics, this is acceptable — a student's weekly accuracy trend does not need to update to the second.

The first-run problem:

REFRESH MATERIALIZED VIEW CONCURRENTLY requires the MV to be populated and have a unique index. On first run (empty MV, no data), you must use plain REFRESH MATERIALIZED VIEW. Subsequent refreshes use CONCURRENTLY, which does not block readers. This is a documented gotcha because forgetting it causes the CONCURRENTLY refresh to fail on an empty MV.

Staggered refresh to avoid spikes:

Six MVs are refreshed every 2 hours. If all six refresh simultaneously, they all run expensive aggregation queries on user_attempts at the same time. The database CPU spikes.

AttemptIQ staggers the refreshes:

// workers/analytics-refresh.worker.ts
const MV_SCHEDULE = [
  { name: 'mv_user_subject_stats',  delayMinutes: 0  },
  { name: 'mv_leaderboard',         delayMinutes: 10 },
  { name: 'mv_user_trap_stats',     delayMinutes: 20 },
  { name: 'mv_user_pressure_stats', delayMinutes: 30 },
  { name: 'mv_user_monthly_solve',  delayMinutes: 40 },
  { name: 'mv_user_topic_stats',    delayMinutes: 50 },
];

Each MV refreshes 10 minutes apart. The CPU load spreads across the hour instead of spiking at H:00. This is a simple but effective technique — it costs nothing and prevents a periodic performance cliff.

The Mental Model (take this with you)

Denormalization is a deliberate tradeoff, not a mistake. When you know that data will always be read together and that aggregation over it needs to be fast, storing it redundantly in the read path is correct. The cost is maintaining the invariant at write time. Indexes are a sorted copy of your data, optimized for a specific access pattern. They speed reads at the cost of write overhead. And materialized views are pre-computed query results — they trade freshness for speed.

Chapter 6: API Design

The Contract Between Frontend and Backend

The Problem First

An API is a contract. The frontend depends on it. If the backend changes the contract without warning, the frontend breaks. If the contract is inconsistent — some routes return data.results, others return items, others return the payload directly — every frontend developer has to read the source to know what to expect.

The Insight

Consistency is more important than perfection. A consistent API with slightly imperfect naming is vastly better than an inconsistent API with beautiful naming. The entire frontend can be written by a single developer if they can predict the shape of every response without checking the source.

The Solution AttemptIQ Uses

Every successful response follows one envelope:

{
  "data": { ...resource }
}

Every error follows one envelope:

{
  "error": "Human-readable message",
  "code": "MACHINE_READABLE_CODE",
  "details": {}
}

The machine-readable code allows client-side logic to branch without string-matching error messages. The message can change for localization; the code must not change.

// lib/errors.ts — the entire error system
export class AppError extends Error {
  constructor(
    message: string,
    readonly status: number,
    readonly code: string,
  ) { ... }
}

export class SessionAlreadySubmittedError extends AppError {
  constructor(message = 'Session already submitted') {
    super(message, 409, 'SESSION_ALREADY_SUBMITTED');
  }
}

// In route handlers:
export function toResponse(err: unknown): Response {
  if (err instanceof AppError) {
    return Response.json({ error: err.message, code: err.code }, { status: err.status });
  }
  console.error('[Unhandled error]', err);
  return Response.json({ error: 'Internal server error', code: 'INTERNAL_ERROR' }, { status: 500 });
}

Every service throws typed errors. Every route handler catches with toResponse(). The HTTP status code is embedded in the error class. You cannot accidentally return 200 for an error — the error itself carries the status.

Why REST?

AttemptIQ uses REST-style routes — not GraphQL, not gRPC, not tRPC. The decision is pragmatic:

• REST is the default for Next.js API routes
• The operations are simple CRUD — create session, submit attempt, get analytics
• There are fewer than 50 API endpoints — no need for a schema language
• REST is universally understood; any frontend developer can reason about it

GraphQL's benefit — fetching exactly the shape you need — only matters when your frontend needs are highly variable or when you are building a public API for third parties. Neither applies here.

Request Validation — Why Zod?

Every route handler validates input with Zod before doing anything else. Invalid input → 400 immediately. No business logic runs on invalid data.

// app/api/attempts/submit/route.ts
const submitSchema = z.object({
  questionId: z.string().uuid(),
  sessionId: z.string().uuid(),
  selectedAnswer: z.enum(['A', 'B', 'C', 'D']).nullable(),
  solveMethod: z.enum(['knowledge', 'elimination', 'educated_guess', 'pure_guess']).nullable(),
  optionsEliminated: z.array(z.enum(['A', 'B', 'C', 'D'])).max(3).nullable().default(null),
  timeTakenSeconds: z.number().int().min(0).max(7200),
});

Why Zod specifically? Because it generates TypeScript types from schemas, so the validated data is fully typed without manual type declarations. parsed.data in TypeScript code is type-safe — no runtime surprises.

Why validate at the API boundary and not in the service? Because the service should not know about HTTP. Services receive typed DTOs. If the input is invalid at the API boundary, the service never sees it. The service can assume its inputs are valid — it focuses on business logic, not input hygiene.

SQL injection is impossible — here is why:

Every database query uses parameterized statements:

await db.query(
  `SELECT * FROM sessions WHERE id = $1 AND user_id = $2`,
  [sessionId, userId],
);

The $1 and $2 are placeholders. The values are passed separately to the PostgreSQL driver, which handles escaping. The SQL string is never concatenated with user input. An attacker who sends sessionId = "'; DROP TABLE sessions; --" sees their string treated as a literal value, not SQL. The query fails with "invalid UUID" before it even reaches the database.

There is one exception: materialized view names cannot be parameterized in PostgreSQL. The analytics worker handles this by whitelisting:

// workers/analytics-refresh.worker.ts
const ALLOWED_VIEWS = MV_SCHEDULE.map((mv) => mv.name);

if (!ALLOWED_VIEWS.includes(viewName)) {
  throw new Error(`Invalid materialized view: ${viewName}`);
}
const sql = `REFRESH MATERIALIZED VIEW CONCURRENTLY ${viewName}`;

The MV name is interpolated — but only after confirming it is in the whitelist. This is the correct approach when parameterization is impossible.

The Thin Route Handler Pattern

Route handlers in AttemptIQ are never longer than ~20 lines. They do exactly three things: authenticate, validate, call a service method. No business logic. No database calls. No conditional branches beyond the error path.

// app/api/sessions/route.ts — entire route handler
export async function POST(req: Request) {
  const session = await auth.api.getSession({ headers: await headers() });
  if (!session) return Response.json({ error: 'Auth required' }, { status: 401 });

  const parsed = createSessionSchema.safeParse(body);
  if (!parsed.success) return Response.json({ error: 'Invalid request' }, { status: 400 });

  const { rows } = await db.query<{ plan_tier: string }>(
    `SELECT COALESCE(plan_tier, 'free') AS plan_tier FROM user_profiles WHERE user_id = $1`,
    [session.user.id],
  );
  const isFree = (rows[0]?.plan_tier ?? 'free') === 'free';

  const sessionId = await sessionsService.createPracticeSession(session.user.id, parsed.data, isFree);
  return Response.json({ data: { id: sessionId } }, { status: 201 });
}

Why keep route handlers thin? Because thin route handlers are:

• Testable: you can test the service independently of HTTP
• Readable: the intent is immediately clear
• Consistent: every route follows the same pattern

The service sessionsService.createPracticeSession() contains all the complexity. The route handler just orchestrates.

The Mental Model (take this with you)

An API is a contract. Consistency is the most important property of that contract. Validate at the boundary. Throw typed errors. Return consistent envelopes. Never return 200 for an error. These rules sound bureaucratic until the first time they prevent a silent failure that costs an hour of debugging.

Chapter 7: Failure and Resilience

What Happens When Things Break

The Problem First

Every external dependency will fail. Not sometimes. Not rarely. Always, eventually. The database will be overloaded. Redis will lose its connection. The payment provider's API will time out. The email service will bounce.

The question is not "will things fail?" It is "when things fail, what does the user experience?"

There are two categories of failure: loud failures (the system returns an error and the user can retry) and silent failures (the system appears to succeed but the work is lost or corrupted). Silent failures are vastly worse. A user who gets an error knows to try again. A user whose email was silently dropped never knows they missed it.

The Insight

Design for failure at every layer. For each external call: what is the timeout? Will we retry? What is the fallback? When a service is down, what does the user see?

The Solution AttemptIQ Uses

The resilience matrix from the docs captures the decision for every external service:

Why no retry on PostgreSQL?

PostgreSQL is the critical path. If the database is down, retrying the same query will fail again. The error should surface immediately so the monitoring system detects it and pages on-call. Hiding database failures with retries masks the most severe class of infrastructure problem.

Why no retry on Razorpay order creation?

Creating a payment order is non-idempotent. Retrying might create duplicate orders. The user would see multiple charges. The correct response to a Razorpay timeout is to tell the user "payment service unavailable, try again" — and let the user decide whether to retry. The server should not retry on their behalf.

The rate limiter fails open:

// lib/rate-limit.ts
} catch (err) {
  // Redis failure — fail open so the site stays up
  try {
    const Sentry = await import('@sentry/nextjs');
    Sentry.captureException(err, {
      tags: { component: 'rate-limiter', identifier },
      level: 'error',
    });
  } catch { ... }
  return { allowed: true, remaining: limit, resetInSeconds: windowSeconds };
}

When Redis is down, the rate limiter allows all requests. This is "fail open." The alternative — fail closed, blocking all requests — would take the site down along with Redis. A brief gap in rate limiting is far less harmful than a site-wide outage. But the failure is captured in Sentry so on-call knows Redis is unhealthy.

The cache-with-fallback pattern:

// lib/cache/analytics.cache.ts
export async function getCachedSubjectStats(userId: string) {
  const key = `analytics:subject_stats:${userId}`;
  try {
    const cached = await redis.get(key);
    if (cached) return JSON.parse(cached);
  } catch {
    // Redis down — fall through to DB
  }

  const data = await db.query(
    'SELECT * FROM mv_user_subject_stats WHERE user_id = $1', [userId]
  );

  try {
    await redis.set(key, JSON.stringify(data.rows), 'EX', 600);
  } catch {
    // Redis down — return data without caching
  }

  return data.rows;
}

This pattern wraps every Redis operation in try/catch. A Redis failure silently degrades to a database query — slower, but correct. The try/catch on the cache write means a Redis failure during write does not prevent the data from being returned.

The BullMQ retry pattern:

Emails are sent via BullMQ queues, not directly from the API route. This means:

1. The API responds immediately ("welcome email queued")
2. BullMQ retries the email up to 3 times with exponential backoff
3. If all retries fail, the job enters the "failed" set
4. The admin can inspect and retry from the Bull Board UI

// lib/queues.ts
const defaultJobOptions = {
  attempts: 3,
  backoff: { type: 'exponential' as const, delay: 5000 },
  removeOnComplete: { count: 100 },
  removeOnFail: { count: 500 }, // keep failed jobs for debugging
};

removeOnFail: { count: 500 } keeps the last 500 failed jobs in Redis. This is the dead-letter queue — not a separate queue, but a set of failed jobs that can be inspected and retried.

The health check endpoint:

// app/api/health/route.ts
const checks: Record<string, boolean> = {};
await db.query('SELECT 1').then(() => checks.db = true).catch(() => checks.db = false);
await redis.ping().then(() => checks.redis = true).catch(() => checks.redis = false);

const ok = Object.values(checks).every(Boolean);
return Response.json({ ok, checks }, { status: ok ? 200 : 503 });

UptimeRobot pings /api/health every 5 minutes. If the response is 503, it sends an email and SMS alert. This is the outermost monitoring layer: if anything critical is down, the developer knows within 5 minutes.

The health endpoint requires a secret token to return details — without it, only { ok: true/false } is returned. This prevents infrastructure reconnaissance by external attackers who probe the health endpoint to learn what internal services exist.

The Redis allkeys-lru gotcha:

Redis has a documented bug in the infrastructure (docs/gotchas.md #34). The original configuration used maxmemory-policy allkeys-lru. This policy evicts any key when memory is full — including BullMQ job keys that have no TTL. A job could silently vanish mid-queue with no error, no log, and no retry.

The fix is volatile-lru: only evict keys that have a TTL set. Cache keys have TTLs and are safe to evict. BullMQ keys have no TTL and must never be evicted.

redis-cli CONFIG SET maxmemory-policy volatile-lru
redis-cli CONFIG REWRITE

This is the difference between a bug that causes occasional silent data loss (catastrophic) and one that causes an occasional cache miss (acceptable). Never use allkeys-* eviction policies when Redis serves both a cache and a job queue.

The deployment — PM2's restart rule:

Every deployment kills the old Node.js processes and starts new ones. The critical rule is: never use pm2 restart or pm2 reload — both reuse stale environment variables from when PM2 was first started.

# CORRECT — forces fresh environment variable pickup
pm2 stop attemptiq && pm2 delete attemptiq && pm2 start ecosystem.config.js && pm2 save

This was discovered the hard way: after updating the .env file on the VPS, pm2 reload left the old env vars in memory. The new env vars were never picked up until the process was fully stopped and deleted.

The Mental Model (take this with you)

Every external call is a potential failure. Design the failure mode first, then the success path. For each external dependency, ask: what is the timeout? What is the retry strategy? What does the user see when it fails? Is this failure loud or silent? Silent failures are always worse. Fail loudly, degrade gracefully, and make sure monitoring catches it within minutes.

Chapter 8: Performance

Where the Codebase Is Fast and Where It Is Slow

The Problem First

Performance is not a single property. It has two dimensions: latency (how long does one request take?) and throughput (how many requests per second can the system handle?). A system can have excellent latency but poor throughput, or vice versa.

For AttemptIQ, latency matters most: a student waiting 3 seconds for their analytics to load has a bad experience. Throughput matters for the system's ceiling: at what point does adding more users cause everyone's latency to degrade?

The Insight

Performance problems fall into three categories:

1. CPU-bound: the server is computing (rendering HTML, running aggregations)
2. IO-bound: the server is waiting (database query, external API call)
3. Bandwidth-bound: the network is saturated (large payloads, many requests)

Most web application performance problems are IO-bound. The fix for IO-bound problems is caching — storing the result of a slow IO operation in a fast storage layer.

Cloudflare Edge Cache — Zero Load on Origin

Public question pages (/upsc/[subject]/[year]/[slug]) are served to anyone — no auth required, no personalization. These pages are perfect candidates for edge caching.

When a student visits /upsc/history/2023/which-mughal-emperor-built-the-red-fort, the HTML is rendered by Next.js on the first request and cached at Cloudflare's edge. Every subsequent request for the same URL from anywhere in the world gets the cached HTML — the VPS is never hit.

# Cache-Control for public question pages
Cache-Control: public, max-age=3600, s-maxage=3600

When an admin approves a question or edits its content, the old cached page must be invalidated:

// lib/cloudflare/purge.ts
export async function purgeCloudflareCache(urls: string[]) {
  await fetch(
    `https://api.cloudflare.com/client/v4/zones/${process.env.CF_ZONE_ID}/purge_cache`,
    {
      method: 'POST',
      headers: { Authorization: `Bearer ${process.env.CF_API_TOKEN}` },
      body: JSON.stringify({ files: urls }),
    }
  );
}

This is the cache invalidation problem — one of computer science's two hard problems. The solution here is event-driven purging: when data changes, invalidate the cache for the affected URLs. The alternative (TTL-only expiry) would mean stale content for up to 1 hour after an admin update.

Redis Application Cache — Pre-computed Analytics

Analytics data is expensive to compute (complex aggregations over large tables) but changes slowly (updated every 2 hours via MV refresh). It is a perfect candidate for Redis caching.

// Analytics request path (with Redis hit):
// 1. Read from Redis → ~1ms
// 2. Return data

// Analytics request path (Redis miss, DB fallback):
// 1. Redis miss → ~1ms
// 2. Query mv_user_subject_stats WHERE user_id = $1 → ~5ms
// 3. Write result to Redis with 10-minute TTL → ~1ms
// 4. Return data → total ~7ms

// Without caching (raw query at request time):
// 1. Complex aggregation over user_attempts → ~200–500ms

The difference between serving from Redis (~1ms) and serving from raw aggregation (~500ms) is a 500x improvement in latency for analytics. For the student, this means clicking "Analytics" and seeing their data instantly instead of waiting half a second.

The TTL strategy:

Different data has different freshness requirements:

• Leaderboard: 10 minutes (stale by 10 minutes is fine)
• Analytics per user: 10 minutes (updated by MV refresh every 2 hours)
• Platform config (pricing): 24 hours (rarely changes)

A cache entry must expire before it becomes misleading. If a user just finished a mock test and refreshes their analytics, seeing data from 10 minutes ago is acceptable. Seeing data from yesterday is confusing.

pgBouncer — Multiplexing Connections

Each Next.js process (PM2 runs 4 cluster instances) opens a pool of database connections. Without connection pooling, 4 instances × 10 connections = 40 direct PostgreSQL connections. PostgreSQL allocates resources (memory, a process on older versions) for each connection. At 200 direct connections, PostgreSQL starts degrading.

pgBouncer sits between Next.js and PostgreSQL. It accepts many connections from the application and maintains a small pool of real PostgreSQL connections:

4 Next.js instances × 10 pool connections = 40 connections to pgBouncer
pgBouncer → 25 real PostgreSQL connections

40 application connections are multiplexed onto 25 database connections. This works because most requests are not actively using their database connection at any given moment — they are computing, waiting for network, rendering HTML. The connection is idle. pgBouncer lends idle connections to other requests that need them.

The limit of 25 PostgreSQL connections handles 1,000–2,000 concurrent users because at any moment, only a small fraction of users are in the middle of a database query.

PM2 Cluster Mode — Parallelizing CPU Work

Node.js is single-threaded. On a 4-vCPU server, a single Node.js process uses exactly 1 CPU. The other 3 sit idle. PM2 cluster mode spawns 4 Node.js processes (one per CPU), and Nginx's load balancing distributes requests across them.

// ecosystem.config.js
{
  instances: 'max',    // spawn one process per CPU
  exec_mode: 'cluster',
}

This 4x parallelization handles 4x the throughput. Next.js Server Component rendering is CPU-bound (it generates HTML). The cluster multiplies rendering capacity linearly with CPU count.

The ISR gotcha:

Next.js Incremental Static Regeneration (ISR) allows public pages to be re-rendered in the background after their TTL expires, serving the cached version in the meantime. A public question page with revalidate: 3600 is generated once and served from the cache for 1 hour. Then it is silently regenerated in the background.

There is a critical rule: do not read searchParams in a Server Component that uses ISR. Reading searchParams in a Next.js 14 App Router Server Component disables ISR for that page — it becomes fully dynamic, re-rendering on every request. This silently removes the cache and makes every request hit the origin server.

// WRONG — disables ISR
export default async function Page({ searchParams }: { searchParams: { year?: string } }) {
  const data = await fetchSomething(searchParams.year); // ISR now broken
}

// CORRECT — ISR intact
export default async function Page({ params }: { params: { slug: string } }) {
  const question = await questionsRepo.findBySlug(params.slug);
  // searchParams handled in a 'use client' component
}

The output: 'standalone' optimization:

Next.js's standalone output mode reduces the memory footprint of each PM2 instance by approximately 60%. Instead of bundling the entire node_modules directory into the build, it traces exactly which dependencies are needed and bundles only those.

// next.config.mjs
const nextConfig = {
  output: 'standalone',
};

With 4 PM2 instances, each using ~300MB in standalone mode, the total Next.js memory footprint is ~1.2GB. Without standalone mode, each instance would consume ~750MB, totaling ~3GB — more than a third of the VPS's 8GB RAM. This single configuration option doubles the headroom before the server needs to be upgraded.

The Analytics Database Queries — Bounded Aggregations

Some analytics — Consistency Index, Overconfidence Report, Pressure Stats — query raw user_attempts rather than materialized views. Why? Because they compute metrics that are too user-specific to pre-aggregate into a per-user MV without making the MV enormous.

The safety rule: every raw-table analytics query must include WHERE user_id = $1. Without this WHERE clause, the query aggregates all users' attempts — potentially millions of rows. With it, the query is bounded to one user's attempts, which might be a few thousand rows at most.

// analytics.repo.ts — Overconfidence Report
await db.query(
  `SELECT ... FROM user_attempts ua
   WHERE ua.user_id = $1 AND ua.selected_answer IS NOT NULL`,
  [userId],
);

The idx_attempts_user_q: (user_id, question_id) index makes this fast — PostgreSQL can use the index to jump directly to this user's rows.

Latency vs Throughput — AttemptIQ Examples

Latency: the time from the student clicking "Submit" to seeing the result. Target: < 300ms (P95). This is dominated by: network round trip (~20ms), Nginx overhead (~1ms), auth session check (~5ms), database INSERT (~10ms). Total: ~50ms of actual work, plus network. Well within target.

Throughput: how many concurrent students can take tests simultaneously? With 4 PM2 instances and pgBouncer multiplexing 40 connections onto 25 real DB connections, the bottleneck is the database under sustained load. The docs predict 1,000–2,000 concurrent users on the current VPS configuration.

The Mental Model (take this with you)

Caching is the primary performance lever in IO-bound systems. The three-layer cache (Cloudflare edge → Redis → PostgreSQL) ensures that the most expensive operation (database query) is rarely necessary. Parallelism is the primary lever for CPU-bound systems. PM2 cluster mode exploits all CPU cores. Connection pooling is the lever for connection-bound systems. pgBouncer prevents database connections from becoming the bottleneck long before the database itself saturates.

If this were 10x scale

The first bottleneck at 10x scale would be the database CPU, not connections. mv_user_subject_stats refreshes every 2 hours by doing a full aggregation over user_attempts. At 10x the data, this refresh takes 10x longer. The fix: read replicas for analytics queries, more granular refresh scheduling, or partitioned user_attempts by month.

Chapter 9: The Payment System

How Money Moves Through AttemptIQ Safely

The Problem First

Payments are the only place where bugs have direct financial consequences. A bug that causes a student to be charged twice is a refund, a support ticket, and a reputational hit. A bug that activates a plan without payment is revenue loss and potential fraud. A bug that fails to activate after successful payment is the worst: the student paid, gets nothing, and is furious.

The Insight

A payment system is fundamentally a distributed transaction problem. Two systems (Dodo Payments and AttemptIQ's database) must agree on the state of a payment. When one system's network is unreliable, they can disagree. The solution is to design the system so that disagreement is detectable and recoverable, and that every operation is idempotent.

The Solution AttemptIQ Uses

The flow:

1. Student clicks "Buy Annual Plan"
2. POST /api/payments/create-order
   → creates payments row (status: 'pending')
   → creates Dodo checkout session
   → returns checkout URL
3. Student is redirected to Dodo's hosted checkout
4. Student completes payment on Dodo's side
5. Dodo sends webhook to POST /api/webhooks/dodo
6. Webhook handler:
   a. Verifies HMAC signature
   b. Checks idempotency (webhook_events table)
   c. Activates payment + upgrades plan in one atomic transaction
7. Client polls /api/payments/status
8. Client shows success screen

The double-click guard:

What if the student clicks "Buy" twice rapidly? Two order-creation requests race. The guard:

// payments.service.ts — createOrder
const recent = await this.paymentsRepo.findRecentPending(userId, 2); // within 2 minutes
const recentPlanId = (recent?.metadata as Record<string, unknown> | null)?.planId;
if (recent && recentPlanId === planId) {
  // Reuse the existing checkout session
  return { url: existingSession.checkoutUrl, paymentId: recent.id };
}

If a pending order for the same plan exists within 2 minutes, return the same checkout URL. No duplicate orders.

The webhook idempotency problem:

Dodo retries failed webhooks multiple times over 24 hours. If your webhook handler is not idempotent — if processing the same event twice has side effects — a student could be upgraded twice or have duplicate notifications.

AttemptIQ uses a two-layer guard:

Layer 1: webhook_events table

-- webhook-events.repo.ts — insertEvent
WITH ins AS (
  INSERT INTO webhook_events (id, gateway, event_type, payload, processed)
  VALUES ($1, $2, $3, $4, false)
  ON CONFLICT (id) DO NOTHING
  RETURNING true AS is_new, false::boolean AS processed
)
SELECT is_new, processed FROM ins
UNION ALL
SELECT false AS is_new, processed
FROM webhook_events
WHERE id = $1
  AND NOT EXISTS (SELECT 1 FROM ins)

This single atomic statement handles three cases:

• First time we see this event: INSERT succeeds → returns is_new = true → 'inserted'
• Event exists, not yet processed: INSERT conflicts → falls to UNION ALL → returns is_new = false, processed = false → 'duplicate'
• Event exists, already processed: INSERT conflicts → falls to UNION ALL → returns is_new = false, processed = true → 'already_processed'

PostgreSQL serializes concurrent inserts with the same id. If two webhook retries arrive simultaneously, one blocks while the other commits, then reads the committed state. No race window.

Why use this CTE pattern instead of ON CONFLICT DO UPDATE? Because ON CONFLICT DO UPDATE always writes a new tuple, and the xmax = 0 trick for detecting whether it was a fresh insert or an update is unreliable on newer PostgreSQL versions. The CTE pattern has no such ambiguity.

Layer 2: UPDATE WHERE status = 'pending' RETURNING id

Even if Layer 1 were somehow bypassed, the payment activation query guards again:

// payments.service.ts — handleWebhook
const { rows: paymentRows } = await client.query<{ id: string }>(
  `UPDATE payments
   SET dodo_payment_id = $2, status = 'captured', captured_at = now(), ...
   WHERE id = $1 AND status = 'pending'
   RETURNING id`,
  [paymentId, dodoPaymentId, totalAmount],
);

if (paymentRows.length === 0) {
  await client.query('ROLLBACK');
  return { status: 'already_processed' };
}

WHERE status = 'pending' means this UPDATE only succeeds once — when the payment transitions from pending to captured. A second attempt sees 0 rows and rolls back.

The atomic transaction:

await client.query('BEGIN');

// 1. Capture payment — guards against double-processing
// 2. Upgrade plan — atomically linked to payment capture
await client.query(
  `UPDATE user_profiles
   SET plan_tier = $3, plan_expires_at = GREATEST(COALESCE(plan_expires_at, now()), now()) + make_interval(days => $2)
   WHERE user_id = $1`,
  [metaUserId, planDays, planId],
);
// 3. Mark webhook processed — inside same transaction
await client.query(`UPDATE webhook_events SET processed = true WHERE id = $1`, [dodoPaymentId]);

await client.query('COMMIT');

All three updates in one transaction. If the server crashes between the plan upgrade and COMMIT, PostgreSQL rolls back everything. Dodo retries the webhook. The transaction runs again, this time to completion.

Critically, webhook_events.processed = true is set inside the same transaction. Previously (in an earlier version), it was set after COMMIT. A server crash between COMMIT and the markProcessed() call would leave processed = false, causing the event to be processed again on the next retry. Moving it into the transaction eliminates this gap.

Safety guarantees:

Double charge: structurally impossible. Access without payment: structurally impossible. Payment lost: Dodo retries until success.

The HMAC signature check:

// Dodo webhook verification — first thing the handler does
const event = verifyDodoWebhook(rawBody, headers);

The HMAC signature verifies that the webhook payload was sent by Dodo, not by an attacker crafting a fake payment confirmation. An invalid signature must return 400, never 200. Returning 200 to an invalid signature would acknowledge a forged webhook — an attacker could send fake "payment succeeded" events and upgrade their account without paying.

The raw body must be read as text, not parsed as JSON, before verification. HMAC is computed over the raw string. Re-serializing JSON loses whitespace and key ordering, producing a different string and a different hash — signature verification would fail for legitimate webhooks.

The reconciliation safety net:

Every 30 minutes, a BullMQ cron job runs payment-recon. It queries Dodo's API for recent successful payments and cross-checks the payments table. If a payment shows as captured in Dodo but is still pending in the database — the webhook was lost or failed — the reconciliation job activates it.

This is the defense against webhook delivery failures. The webhook handler is the fast path; the reconciliation cron is the safety net.

Chapter 10: Background Jobs

Why Async, Why Not Synchronous

The Problem First

Some operations are slow (analytics refresh), some are unreliable (email delivery), and some run on a schedule (daily revision cache generation). None of these belong in a request-response cycle.

The Insight

Any operation with uncertain duration or reliability should be decoupled from the request that triggers it. The request should enqueue the work and return immediately. The work runs independently, with retries if it fails, and is monitored for health.

The Solution AttemptIQ Uses

BullMQ + Redis. BullMQ is a job queue library that uses Redis as its backing store. Jobs are serialized to Redis, picked up by workers, executed, and removed on success (or moved to the failed set on exhaustion of retries).

// lib/queues.ts — the seven queues
export const analyticsRefreshQueue = new Queue('analytics-refresh', { connection, defaultJobOptions });
export const revisionListQueue     = new Queue('revision-list', { connection, defaultJobOptions });
export const scheduledReviewsQueue = new Queue('scheduled-reviews', { connection, defaultJobOptions });
export const emailQueue            = new Queue('email', { connection, defaultJobOptions });
export const paymentReconQueue     = new Queue('payment-recon', { connection, defaultJobOptions });
export const dbBackupQueue         = new Queue('db-backup', { connection, defaultJobOptions });
export const monthlyReportQueue    = new Queue('monthly-reports', { connection, defaultJobOptions });

Each queue has a dedicated purpose. They are not conflated. An email job and an analytics refresh job are isolated — a bug in one queue does not affect the other.

The ioredis version mismatch gotcha:

BullMQ bundles its own version of ioredis. The application also uses ioredis for the Redis cache client. These two versions can be incompatible at the TypeScript type level. The solution: never pass the application's ioredis instance to BullMQ. Always pass a plain connection config object:

// WRONG — type mismatch between ioredis versions
const worker = new Worker('email', handler, { connection: redis });

// CORRECT — plain config, BullMQ uses its own ioredis internally
const connection = {
  host: process.env.REDIS_HOST ?? '127.0.0.1',
  port: parseInt(process.env.REDIS_PORT ?? '6379', 10),
  maxRetriesPerRequest: null as null, // required by BullMQ
};
const worker = new Worker('email', handler, { connection });

maxRetriesPerRequest: null is specifically required by BullMQ. ioredis defaults to 3; BullMQ needs it set to null because it manages retries at the job level, not the Redis command level.

The cron_health protocol:

Every job writes to cron_health on completion:

// workers/analytics-refresh.worker.ts
await db.query(
  `INSERT INTO cron_health (job_name, last_run_at, last_status, error_message)
   VALUES ($1, now(), $2, $3)
   ON CONFLICT (job_name) DO UPDATE
   SET last_run_at = now(), last_status = $2, error_message = $3`,
  [jobName, success ? 'success' : 'failed', errorMessage],
);

ON CONFLICT DO UPDATE means this is an upsert — it creates the row on the first run, updates it on every subsequent run. The /api/health endpoint reads cron_health and returns 503 if any job has failed twice consecutively. UptimeRobot catches this and alerts.

Idempotent workers:

A cron job can fire twice due to Redis failover or clock skew. Every worker must be idempotent — running it twice produces the same result as running it once. The REFRESH MATERIALIZED VIEW CONCURRENTLY operation is idempotent: refreshing twice gives the same final MV state.

The batching rule for large user operations:

The revision list worker must generate personalized revision lists for all active users. Looping over all users in one go would:

• Hold database connections for too long
• Consume unbounded memory
• Block other jobs from running

The solution is pagination:

// docs/jobs.md — revision list batching pattern
let offset = 0;
while (true) {
  const users = await db.query(
    'SELECT id FROM user_profiles WHERE is_deleted = FALSE ORDER BY id LIMIT 500 OFFSET $1',
    [offset]
  );
  if (users.rows.length === 0) break;
  for (const user of users.rows) {
    await generateRevisionList(user.id);
  }
  offset += 500;
}

Process 500 users, commit and move on. Each batch is bounded in time and memory. If the worker crashes halfway, it will resume from offset 0 on the next run — slightly redundant, but correct.

Chapter 11: The Gaps

A Brutally Honest Assessment of What Is Missing

This chapter does not pretend the system is complete. Every production system has gaps. The difference between a junior engineer and a senior engineer is that the senior engineer can see the gaps and prioritize them.

Gap 1: No Circuit Breaker

What is missing: When an external service like Redis or Razorpay is flapping (failing intermittently), the system retries every request against the failing service. Each retry adds latency. At high request rates, the backlog of retrying requests consumes thread pool slots and memory. This is the cascading failure pattern — the flapping dependency causes the entire application to degrade.

A circuit breaker monitors failure rates. When failures exceed a threshold, it "opens the circuit" — subsequent requests fail immediately rather than being attempted. After a timeout, the circuit enters "half-open" state and allows a test request through. If it succeeds, the circuit closes. If not, it stays open.

What breaks: Without a circuit breaker, Redis going up and down 10 times per minute causes 10 × N retry storms, where N is the number of concurrent requests. On a busy server, this can cause request latency to spike from 100ms to 5s+ as threads pile up waiting for retries.

Simplest fix: Add the opossum circuit breaker library for Redis and Razorpay calls. Configure it to open the circuit after 50% failure rate in a 10-second window.

Production-grade fix: Expose circuit breaker state in the /api/health endpoint. Alert when a circuit opens. Pre-warm the circuit after services recover.

The docs acknowledge this gap: docs/resilience.md explicitly marks circuit breakers as "Future — Not v1."

Gap 2: Sessions Table Grows Unbounded

What is missing: The sessions table grows by one row every time a student starts a practice session or mock test. There is no archival cron job.

What breaks: At 100,000 sessions, SELECT * FROM sessions WHERE user_id = $1 AND is_submitted = false is slower — the partial index idx_sessions_active helps, but the table itself occupies more disk space and more of PostgreSQL's buffer pool.

The session archival query:

DELETE FROM sessions
WHERE is_submitted = TRUE AND created_at < now() - interval '90 days';

This is safe only after confirming user_attempts rows exist — the attempts table is the analytics source of truth. Sessions can be deleted; attempts cannot. The query is documented in docs/database.md but not yet implemented as a scheduled job.

Simplest fix: Add a daily BullMQ cron job that runs the archival DELETE at 3am.

Gap 3: No Formal Test for Webhook Idempotency

What is missing: The webhook idempotency mechanism — the two-layer guard using webhook_events CTE and WHERE status='pending' — is sophisticated and correct. But it is not covered by the test suite (__tests__/payments.service.test.ts exists but the concurrent retry scenario is not exercised).

What breaks: A future refactor of the webhook handler might accidentally break the idempotency guarantee. Without tests for the concurrent-retry scenario, this break would be silent until a production incident.

Simplest fix: Integration tests that simulate two concurrent webhook requests for the same payment ID and assert that the plan is upgraded exactly once.

Gap 4: Subject Leaderboard Hits Raw Attempts Table

What is missing: The global leaderboard uses mv_leaderboard — a pre-computed materialized view. Fast. But the subject-specific leaderboard (top 100 in History, for example) has no MV:

// analytics.repo.ts — getSubjectLeaderboard
// mv_leaderboard is global; subject leaderboard queries raw attempts
const { rows } = await db.query(
  `SELECT ...
   FROM user_attempts ua
   JOIN questions q ON q.id = ua.question_id
   LEFT JOIN user_profiles up ON up.user_id = ua.user_id
   WHERE q.subject = $1 ...
   GROUP BY ua.user_id, up.display_name
   HAVING COUNT(*) >= 10
   ORDER BY accuracy DESC
   LIMIT $2`,
  [subject, limit],
);

At scale, this is a multi-table join with aggregation over potentially millions of rows. No MV protects it.

What breaks: At 100,000+ attempts per subject, the subject leaderboard page load time degrades from milliseconds to seconds.

Simplest fix: Add Redis caching for subject leaderboard results with a 10-minute TTL. This converts the slow query to a Redis hit for 99%+ of requests.

Production-grade fix: Add mv_leaderboard_by_subject materialized views, one per subject, refreshed alongside the global leaderboard.

Gap 5: The Redis Port Configuration Bug

What is documented: docs/gotchas.md #20 notes that DATABASE_URL on the VPS was found pointing to PostgreSQL port 5432 instead of pgBouncer port 6432. This was discovered and documented in a real incident.

What this means: When the application bypasses pgBouncer and connects directly to PostgreSQL, it opens direct connections for each pool member. With 4 PM2 instances × 10 connections = 40 direct connections to PostgreSQL, and PostgreSQL configured with max_connections = 100, the application consumes 40% of PostgreSQL's connection budget. Under load, with other maintenance connections, this causes "too many connections" errors.

Current status: This is a known-fixed bug, but it is worth documenting as a gap because the fix lives in an environment variable on the VPS and is not verified in code. There is no application-level check that verifies the database URL points to port 6432 on startup.

Simplest fix: Add a startup assertion:

// lib/db/client.ts — startup check
const url = new URL(process.env.DATABASE_URL!);
if (url.port !== '6432') {
  console.warn('[DB] Warning: DATABASE_URL does not point to pgBouncer (port 6432). Connection pooling is disabled.');
}

Gap 6: No Backpressure on the Ingestion Queue

What is missing: The Python ingestion script processes questions sequentially. If 200 questions are being enriched and the Claude API rate limit is hit, the script sleeps with exponential backoff. During this sleep, nothing else is being processed. There is no way for an operator to see how many questions are queued, how many have succeeded, or how long the estimated completion time is.

What breaks: For large ingestion jobs (500+ questions), the operator has no visibility without reading the log file. If the script crashes mid-run, it must be resumed manually with --replay-failed.

Simplest fix: Move the question enrichment loop into BullMQ jobs. Each question becomes a job. The BullMQ dashboard shows progress, failures, and estimated completion. Jobs retry automatically on Claude API errors.

Production-grade fix: Implement the ingestion pipeline as a proper event-driven workflow with a state machine: extracted → enriched → validated → staged → approved. Each state transition is a separate job with its own retry policy.

Gap 7: The searchParams ISR Footgun

What is documented: docs/gotchas.md #33 documents a subtle Next.js footgun: reading searchParams in a Server Component that uses ISR silently disables ISR for that page.

What this means: A developer adding a filter dropdown to a public question page (/upsc/[subject]) might naturally read searchParams in the Server Component to apply the filter server-side. This disables ISR for the entire page, causing every request to hit the origin server and generating a new HTML response — potentially increasing server load 100x for popular pages.

Current status: Known and documented, but there is no linting rule or type-level enforcement preventing it.

Simplest fix: Add an ESLint rule that warns when searchParams is accessed in a file that also uses export const revalidate or export const dynamic.

Epilogue: The Whiteboard Test

This book set out to teach systems thinking through reverse-engineering. Every chapter started from a problem, exposed the naive failure, arrived at the insight, and showed the real code as proof.

Here is the whiteboard test: given a blank whiteboard and these requirements — exam prep SaaS, UPSC questions, practice sessions, mock tests, analytics, payments, single VPS — can you now sketch the architecture and justify every decision?

Start with the data. There are questions (with secret answers), sessions (a contract of selected questions), and attempts (one per question per session). The attempts table is the source of truth for everything — accuracy, streaks, subject weakness. Denormalize session_type into attempts so analytics can aggregate without joins.

Add the request path. Browser → CDN → Nginx → Next.js → pgBouncer → PostgreSQL. Each layer has one job. CDN absorbs static load. Nginx rate-limits. Next.js renders and validates. pgBouncer multiplexes connections. PostgreSQL stores truth.

Add the async path. Email, analytics refresh, revision generation, payment reconciliation — none of these belong in a request handler. BullMQ + Redis. Producers enqueue jobs. Workers consume them. Retries are automatic. The cron_health table makes every job's health visible.

Add the payments path. Two systems must agree: Dodo says "paid," AttemptIQ must activate the plan. The agreement mechanism is a two-layer idempotency guard: the webhook_events CTE prevents duplicate processing, and the WHERE status='pending' RETURNING id guard prevents duplicate state transitions. Both writes happen in one atomic transaction.

Add the security layers. Parameterized queries make SQL injection impossible. Zod schemas make mass assignment impossible. HMAC verification makes forged webhooks impossible. httpOnly cookies make session theft via XSS impossible. The correct_answer invariant makes answer exposure impossible.

If you can draw this on a whiteboard and explain every choice when challenged — why pgBouncer, why BullMQ over PM2 cron, why volatile-lru not allkeys-lru, why the session limit guard must be atomic, why the webhook CTE avoids the xmax trick — you are thinking like a systems engineer.

That is the goal. Not knowing what AttemptIQ does. Knowing why.

The source of truth for this book is the AttemptIQ codebase itself. All code snippets are copied from the actual implementation. Where something is uncertain or undocumented, this book says so explicitly.