ScoreResume: Systems Thinking From First Principles

A reverse-engineering of every decision in ScoreResume, down to its atomic justification.

Preface

This book does not explain what ScoreResume does. It explains why every decision exists — starting from the problem, not the solution.

The reader is assumed to be intelligent but new to production systems thinking. After reading this book, you should be able to sit in front of a blank whiteboard, hear a product requirement, and reason your way to a production-grade architecture — on your own.

The method is simple: start from the problem. Every time you see a technical decision, ask why until there is nothing left to ask. The insight lives at the bottom of that chain.

ScoreResume is an AI-assisted resume scoring, building, and optimization platform. Users upload or build resumes, receive a deterministic score (0–100) with actionable deductions, download watermark-free PDFs, and get AI-powered rewrites. It runs on a single Hetzner VPS alongside AttemptIQ. Every trade-off in this book is a real trade-off made under bootstrapped constraints.

Chapter 1: Foundation — The System as a Whole

What Is ScoreResume, Exactly?

Before reasoning about any part, you need to see the whole.

ScoreResume is a Next.js 14 App Router monolith. One codebase, one deployment, one process manager. It does not split into microservices. It runs on a bare Linux server in a Hetzner data centre — the same VPS that runs AttemptIQ on a different port.

Here is every moving part, mapped:

Internet
  │
  ▼
Cloudflare (DNS + DDoS + edge cache + CF-IPCountry header)
  │
  ▼
Nginx (SSL termination + rate limiting + reverse proxy)
  │
  ▼
Next.js 14 App Router (PM2 cluster, port 3001)
  ├── App Router Groups
  │     ├── (marketing)/   — landing, pricing, terms — no auth
  │     ├── (app)/         — dashboard, settings, portfolio — requireUser()
  │     ├── (auth)/        — login, signup — no auth
  │     ├── (public)/      — shareable resume/portfolio views — no auth
  │     ├── (admin)/admin/ — admin panel — password-gated
  │     ├── resume/        — builder — requireUser()
  │     └── p/[slug]/      — public portfolio pages
  │
  ├── API Routes (lib/api/ pattern)
  │     requireUser() → withRateLimit() → safeParse() → logic
  │
  ├── lib/
  │     ├── scoring/       — 5-phase deterministic engine
  │     ├── payments/      — DodoPayments, pack lifecycle, tier gate
  │     ├── pdf/           — BullMQ queue + worker
  │     ├── ai/            — Gemini, cost guard, per-feature functions
  │     ├── auth/          — Better Auth, guest sessions
  │     ├── redis/         — rate limiter, BullMQ connection
  │     └── r2/            — Cloudflare R2, signed URLs, key helpers
  │
  └── Prisma ORM → PostgreSQL (port 5432)

Redis (port 6379, localhost only)
  ├── Rate limit sorted sets (sliding window)
  ├── AI concurrency lock (per-user NX key)
  └── BullMQ job queues (PDF generation)

BullMQ Worker (separate PM2 process: scoreresume-worker)
  └── pdf-generation queue → render → upload to R2 → update DB

Cloudflare R2 (S3-compatible object storage)
  ├── watermarked/{userId}/{resumeId}.pdf  — preview PDFs
  └── resumes/{userId}/{resumeId}/{version}.pdf — final PDFs

Two SaaS products share one VPS. ScoreResume is on port 3001; AttemptIQ is on port 3000. They share Nginx but have strictly separate PM2 processes, databases, Redis namespaces, and Nginx configs.

The Request Lifecycle

Let us trace a single request — a user downloading a clean PDF — from button click to file in hand.

Step 1: The browser. User clicks "Download PDF." The client flushes any pending resume saves, then sends:

POST /api/pdf/generate
Cookie: scoreresume.session_token=<token>

{ "resumeId": "uuid", "packId": "uuid" }

Step 2: Cloudflare. Cloudflare terminates TLS, checks for DDoS patterns, and forwards to the VPS with the real client IP in CF-Connecting-IP. It also sets CF-IPCountry: IN (or US, etc.) — this header is used for payment currency detection and cannot be spoofed by the client.

Step 3: Nginx. Nginx receives on port 443, decrypts HTTPS, checks the rate limit zone for /api/, and proxies to localhost:3001.

Step 4: Next.js route handler. The pattern is always the same:

// app/api/pdf/generate/route.ts
const identity = await requireUser()      // 1. Auth — DB lookup on sessions table
const rateLimit = await withRateLimit(…)  // 2. Rate limit — Redis sorted set check
const body = GenerateSchema.safeParse(…)  // 3. Validation — Zod parse
// 4. Business logic

Step 5: Fast path check. If resume.pdfR2Key is already set (cached from previous generation) and the resume hasn't changed, return a fresh signed URL immediately. No BullMQ job, no credit deduction. The user re-downloads the same PDF for free. This fast path collapses the cost of repeat downloads to a single R2 presign operation.

Step 6: Credit deduction path. If no cached PDF exists, call getAvailablePack() to find the oldest active pack with remaining credits (FIFO). Add a BullMQ job with jobId = "pdf:final:{resumeId}". Return { jobId }. The client polls /api/pdf/status/[jobId].

Step 7: The BullMQ worker. The worker picks up the job. It deducts one credit atomically using a Redis idempotency key before rendering (deduct before the expensive work, refund in the catch block — see Chapter 4). It renders the resume to PDF via @react-pdf/renderer, uploads to R2 at a versioned path, updates resume.pdfR2Key in the database, and returns a signed download URL.

Step 8: The client polls. GET /api/pdf/status/[jobId] checks BullMQ job state. On completed, the handler re-generates a fresh signed URL from the stored r2Key (the URL in the job result may have expired during render). Returns { status: "completed", url }.

Total elapsed time from click to download: typically 3–8 seconds (PDF render is the bottleneck).

The mental model: Every request passes through auth → rate limit → validation before any expensive work happens. Auth is not a formality — it is the gate that makes every subsequent check meaningful. Rate limiting after auth means limits are per-user, not per-IP (which would penalise shared corporate networks).

Chapter 2: Identity and Trust

Authentication — The Cookie Is Not the User

The Problem First

HTTP is stateless. Every request arrives knowing nothing about the previous request. A user who proved their identity five seconds ago must prove it again on the next request — or the system must remember the proof for them.

The Insight

The insight is identical to every session system ever built: prove identity once, issue a token, accept the token on subsequent requests. The choice that matters is what the token proves and whether it can be revoked.

What ScoreResume Uses

Better Auth handles session management. On login, Better Auth issues a random session token and stores it in the sessions table. The browser receives a Set-Cookie response:

Set-Cookie: scoreresume.session_token=<random-token>; HttpOnly; Secure; SameSite=Lax; Path=/

On every protected request, the browser sends the cookie automatically. The server looks it up:

// lib/auth/session.ts
const { user } = await getSession()  // validates against DB

HttpOnly prevents JavaScript from reading the cookie — XSS cannot steal it. SameSite=Lax blocks cross-site POST requests — CSRF cannot abuse it. Secure means it only travels over HTTPS — eavesdropping cannot intercept it.

The Two-Layer Auth Check

Middleware does a fast cookie-presence check on the Edge runtime (no Prisma, no database):

// middleware.ts
const sessionCookie =
  request.cookies.get(SESSION_COOKIE_NAME_SECURE) ??
  request.cookies.get(SESSION_COOKIE_NAME)
if (!sessionCookie?.value) {
  return NextResponse.redirect(new URL("/login", request.url))
}

This is not auth — it is a UX gate. It redirects users who obviously have no session without a DB roundtrip. The actual auth check — verifying the token against the database, checking isBanned, setting Sentry user context — happens inside requireUser() in the route handler, which runs in Node.js with full DB access.

Why two layers? Edge runtime cannot run Prisma. Doing real auth in middleware would require an HTTP call back to the API on every page load, doubling latency for every navigation.

The Ban Check

// lib/api/auth-guard.ts
const user = await prisma.user.findUnique({
  where: { id: identity.userId },
  select: { isBanned: true },
})
if (user?.isBanned) throw "NOT_AUTHENTICATED"

The ban check lives in requireUser, not scattered across individual routes. A ban takes effect on the next request, across every route, automatically. No route developer needs to remember to add it.

The Guest Session

Unauthenticated users can build a resume. A guest session cookie (sr_guest_session) is set on first visit. The worker accepts guestSessionId as the owner identity for PDF generation. When a guest creates an account, the hook cleans up both the old (rr_guest_session) and new (sr_guest_session) cookie names — a migration artifact that proves why cookie names must never be assumed stable.

The Open Redirect Guard

After login, the user should be redirected back to where they were:

// middleware.ts
if (isSafeInternalPath(pathname)) {
  loginUrl.searchParams.set("next", pathname)
}
// After login:
const next = searchParams.get("next")
if (next && isSafeInternalPath(next)) router.push(next)
else router.push("/dashboard")

isSafeInternalPath() only allows paths starting with / that do not contain ://. Without this guard, an attacker crafts https://scoreresume.com/login?next=https://evil.com — the user logs in and is redirected to the attacker's phishing page.

The Mental Model

Session auth is a token-lookup problem. The token proves nothing on its own — it is the DB record it points to that carries truth. The two-layer check (Edge cookie presence → Node.js DB validation) is the correct pattern for Next.js App Router: fast UX rejection at the edge, real security in the server.

Chapter 3: Payments — How Money Moves Safely

The Pack Lifecycle

The Problem First

Payments are where bugs have direct financial consequences. Three bad outcomes exist: user charged twice, user gets access without paying, user paid but access never granted. Any of these is a support ticket, a chargeback, or a reputational hit. Some can happen simultaneously in a distributed system.

The Insight

A payment system is a distributed transaction problem. Two systems (Dodo Payments and ScoreResume's database) must agree on whether a payment occurred. When the network is unreliable, they can temporarily disagree. The design must ensure that disagreement is always resolved in one direction: idempotently, in favor of the user who paid.

The Pack as a State Machine

ScoreResume models payment access as a Pack — a bundle of PDF credits with an expiry date. A pack transitions through exactly two states:

pending_<packId>  →  real_payment_id
(awaiting payment)   (payment confirmed)

The paymentId field on Pack doubles as a state indicator. A pending pack has paymentId = "pending_<packId>". An active pack has paymentId = <real Dodo payment ID>. This design encodes state in an existing field — no extra status column, no migration needed to add states.

// lib/payments/pack-service.ts
export async function createPendingPack(params): Promise<string> {
  const packId = crypto.randomUUID()
  await prisma.pack.create({
    data: {
      id: packId,
      paymentId: `pending_${packId}`, // placeholder — overwritten on activation
      gateway: "dodo",
      expiresAt: null,                // calculated at activation, not now
      // ...
    }
  })
  return packId
}

Notice expiresAt: null. If a user starts checkout and returns 3 days later, setting expiresAt at order creation would rob them of 3 days of a pack they haven't paid for yet. Expiry is calculated at activation — from the moment payment is confirmed, not from when the checkout began.

The Full Flow

1. POST /api/payments/create-order
   → Check for recent pending pack (< 2 min) — reuse if exists
   → createPendingPack() — pack row with pending_<packId>
   → createDodoCheckout() — Dodo checkout session with metadata: { packId, userId }
   → Return { checkoutUrl }

2. User completes payment on Dodo's hosted checkout page

3. Dodo fires POST /api/webhooks/dodo (payment.succeeded)
   → Signature verification — HMAC, raw body
   → Idempotency check — WebhookEvent table
   → activatePack() — sets real paymentId, calculates expiresAt
   → Coupon increment + mark processed — one atomic transaction

4. Client polls GET /api/payments/status or listens for redirect

The Double-Click Guard

What if the user clicks "Buy Now" twice in rapid succession?

// app/api/payments/create-order/route.ts
const recentPending = await prisma.pack.findFirst({
  where: {
    userId,
    packType: planId,
    paymentId: { startsWith: "pending_" },
    createdAt: { gt: new Date(Date.now() - 2 * 60 * 1000) }, // 2-minute window
  },
})
if (recentPending) {
  // Patch coupon if it changed, then reuse the same pack
  return ok({ checkoutUrl: newCheckoutUrl, packId: recentPending.id })
}

Within 2 minutes, both requests return the same pack ID. One checkout exists. One pack row exists. The user cannot accidentally create two packs and pay twice.

The Webhook Idempotency Problem

Dodo retries webhooks if the server doesn't respond within 10 seconds. A server restart during activation, or a slow DB, causes Dodo to retry. Two identical payment.succeeded events arrive. If activation runs twice, the user could get two packs.

ScoreResume uses two layers:

Layer 1: WebhookEvent table

// app/api/webhooks/dodo/route.ts
try {
  await prisma.webhookEvent.create({
    data: {
      id: payment.payment_id,   // Dodo's payment ID is unique per payment
      gateway: "dodo",
      eventType: event.type,
      payload: event,
      processed: false,         // set to true AFTER activation succeeds
    }
  })
} catch (e) {
  if (e.code === "P2002") {    // unique constraint violation = already seen
    const existing = await prisma.webhookEvent.findUnique({
      where: { id: payment.payment_id },
      select: { processed: true },
    })
    if (existing?.processed) {
      return new Response("OK", { status: 200 }) // already done — ack and skip
    }
    // processed=false: previous attempt crashed mid-way — fall through to retry
    existingEvent = true
  }
}

The WebhookEvent.id is the Dodo payment ID — unique per payment. The first delivery inserts the row. The second delivery hits the unique constraint (P2002 in Prisma) and reads the processed flag. If processed = true, skip. If processed = false, the previous attempt crashed mid-activation — retry.

Layer 2: activatePack() idempotency

// lib/payments/pack-service.ts
if (!pack.paymentId.startsWith("pending_")) {
  logger.warn({ packId }, "Pack already active — skipping")
  return
}

activatePack checks whether the pack is still in pending state. A real payment ID means activation already succeeded. Return without doing anything. Two calls to activatePack with the same pack produce the same outcome.

The Atomic Transaction

Activation, coupon increment, and marking the webhook processed all happen in one Prisma transaction:

// app/api/webhooks/dodo/route.ts
await activatePack({ packId, paymentId: payment.payment_id, … })

await prisma.$transaction(async (tx) => {
  // 1. Increment coupon usedCount — conditional, never exceeds maxUses
  if (pack.couponId) {
    await tx.coupon.updateMany({
      where: {
        id: pack.couponId,
        OR: [{ maxUses: null }, { usedCount: { lt: couponMaxUses } }],
      },
      data: { usedCount: { increment: 1 } },
    })
  }

  // 2. Store actual Dodo-charged amount (post-coupon, post-tax)
  await tx.pack.update({ where: { id: packId }, data: { amountPaid: payment.total_amount } })

  // 3. Mark webhook processed — same transaction
  await tx.webhookEvent.update({
    where: { id: payment.payment_id },
    data: { processed: true },
  })
})

If the server crashes after activatePack() but before COMMIT, the transaction rolls back. processed remains false. Dodo retries. activatePack() is idempotent — it no-ops on a pack that's already active. The transaction completes. The user gets their pack.

Security: Negative Amount Guard

const totalAmount = payment.total_amount ?? 0
if (totalAmount < 0) {
  logger.error({ paymentId: payment.payment_id, totalAmount }, "SECURITY: negative payment amount")
  return new Response("OK", { status: 200 })
}

A negative amount — however it arrives — is anomalous. Activating a pack with a negative amount would be a revenue leak. Reject it explicitly, log it, and ack so Dodo doesn't retry.

Country Detection

const cfCountry = request.headers.get("CF-IPCountry") ?? countryCode
const isIndia = cfCountry === "IN"
const productId = getDodoProductId(planId, cfCountry)

INR pricing is cheaper than USD pricing. If country came from the request body, a US user could claim countryCode: "IN" and pay ₹99 instead of $5.99. Cloudflare's CF-IPCountry header is set by the CDN based on real IP geolocation and cannot be spoofed by the client. The body value is only a fallback for local development where Cloudflare is absent.

The Mental Model

A payment system is a state machine with two external participants (you and the payment provider) and one source of truth (your database). The design goal is: no matter what order events arrive in — first delivery, retry, concurrent retry — the database ends in exactly one valid state. Idempotency (the webhook event table) handles the "same event twice" case. Atomicity (the transaction) handles the "crash between steps" case. Validation guards handle the "anomalous event" case.

Chapter 4: AI Features — The Intelligence Layer

Why Three Limiting Layers

The Problem First

ScoreResume calls the Gemini API on behalf of users. Gemini charges per token. A single user making repeated AI calls can cost dollars in minutes. At the platform level, a bug, an attack, or a viral moment can burn through the entire monthly budget in hours if nothing stops it.

The Insight

Cost control for AI is a multi-dimensional problem. One limit doesn't cover all attack surfaces. Rate limiting (requests per day) doesn't prevent a single very expensive call. Budget caps ($20/day platform) don't prevent one user from consuming all of it. Tier gates don't prevent a paid user from hammering the API. You need all three layers, independent, each catching a different failure mode.

The Four Limiting Layers

Layer 1: Feature gate — requireTier()
         "You must have at least an Apply pack to use JD Match."
         Catches: free users calling paid features.
         Error: 402 Payment Required.

Layer 2: Session/pack cap — checkStarterAiLimit() / checkInterviewLimit()
         "Starter pack: max 5 AI uses total. Apply pack: max 5 interview sessions."
         Catches: paid users exhausting their pack's included AI quota.
         Error: 402 Payment Required.

Layer 3: Rate limit — withRateLimit() per route key
         "Max 40 bullet rewrites per 24-hour sliding window."
         Catches: rapid fire abuse within a single day.
         Error: 429 Too Many Requests.

Layer 4: Cost budget — checkAiBudget()
         "User cap: $2/day. Platform cap: $20/day."
         Catches: unexpectedly expensive calls, runaway usage.
         Error: 503 Service Unavailable.

Every AI route must call all relevant layers in order:

// Typical AI route
const identity = await requireUser()
await requireTier(identity.userId, "bullet_rewrite")    // Layer 1
await checkStarterAiLimit(identity.userId)               // Layer 2 (for Starter)
const rateLimit = await withRateLimit("AI_BULLET", …)   // Layer 3
if (!rateLimit.allowed) return ApiErrors.RATE_LIMITED()
await checkAiBudget(identity.userId)                    // Layer 4
// Now call Gemini

Skipping checkAiBudget removes the financial safety net entirely. A bug in the prompt that causes Gemini to return 10,000 tokens instead of 200 could exhaust the daily budget on a single user's session.

The Per-User Concurrency Lock

There is a race condition in the budget check. Five concurrent requests from the same user all call checkAiBudget() before any logAiUsage() writes. All five see spend = $0, all five pass, all five call Gemini. The user has effectively bypassed the limit by concurrency.

The fix is a Redis mutex:

// lib/ai/cost-guard.ts
async function acquireAiLock(userId: string): Promise<boolean> {
  const result = await redis.set(`ai:lock:${userId}`, "1", "EX", 60, "NX")
  return result === "OK"   // NX = only set if not exists
}

SET key value EX 60 NX is atomic in Redis. Only one caller gets "OK". Every other concurrent request gets null — the lock already exists — and the budget check throws immediately. The lock expires in 60 seconds automatically if the AI call crashes without releasing it (prevents a stuck user).

The Model Tier

// lib/ai/gemini.ts
export const GEMINI_MODEL_LITE = "gemini-2.5-flash-lite"  // free users
export const GEMINI_MODEL_PRO  = "gemini-2.5-flash"       // paid users

Free users get Flash Lite ($0.10/$0.40 per million tokens). Paid users get Flash ($0.30/$2.50 per million tokens). The difference matters for bullet rewrites — Flash produces noticeably better output for complex technical roles. The split also keeps free usage cheap: if a free user somehow circumvents limits, they're using the cheap model.

Gemini JSON Reliability

Gemini is instructed to return JSON (responseMimeType: "application/json") but occasionally wraps output in markdown fences (```json ... ```). A bare JSON.parse() throws on fenced output:

// lib/ai/gemini.ts
export function safeParseJson<T>(text: string): T {
  const stripped = text
    .replace(/^```(?:json)?\s*/i, "")
    .replace(/\s*```\s*$/i, "")
    .trim()
  return JSON.parse(stripped) as T
}

Every AI call that expects JSON uses safeParseJson. A GeminiOutputError (malformed JSON) is caught in the route handler and returns 503 — "temporarily unavailable, retry" — not 500, which would signal a server bug.

The Retry Pattern

Gemini occasionally returns rate limit errors (429) or transient server errors (500/503). The retry logic uses exponential backoff with jitter:

// lib/ai/gemini.ts — inside callGemini()
for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
  try {
    result = await model.generateContent(userMessage)
    break
  } catch (error) {
    if (isRetryable && attempt < MAX_RETRIES) {
      const delayMs = Math.min(1000 * 2 ** attempt + Math.random() * 500, 8000)
      await new Promise(r => setTimeout(r, delayMs))
      continue
    }
    throw error
  }
}

The jitter (+ Math.random() * 500) prevents multiple server instances from retrying in synchrony — which would create a "retry storm" that worsens the rate limit situation.

The Mental Model

AI cost control is not a single dial. It is a set of independent guards, each protecting against a different failure mode. Tier gates are about entitlement. Session caps are about usage fairness within a paid period. Rate limits are about temporal throttling. Budget caps are the financial circuit breaker. Miss any one layer and you have a hole in the protection.

Chapter 5: The PDF Pipeline — Async Manufacturing

Why PDF Generation Is Not in the Request Handler

The Problem First

A user clicks "Download PDF." The server must render a React component to PDF (CPU-intensive), upload the result to Cloudflare R2 (network I/O), and return a download URL. This takes 3–8 seconds.

The naive implementation: do all of this synchronously in the route handler. The user waits 8 seconds before anything happens. The route handler holds an HTTP connection and a server thread for the entire duration. If the server has 10 concurrent PDF requests, 10 threads are blocked waiting for render + upload to complete. The server becomes unresponsive to all other requests.

The Insight

Slow, fallible, CPU-intensive operations must be decoupled from the request that triggers them. The trigger (POST /api/pdf/generate) should be fast and synchronous. The processing (render, upload) should be asynchronous, retryable, and observable.

The Queue-Based Architecture

POST /api/pdf/generate
  → Fast path: if pdfR2Key is cached, return fresh signed URL (< 50ms)
  → Else: validate pack, deduct intent, addPdfJob()
  → Return { jobId } immediately

Client polls GET /api/pdf/status/[jobId]
  → BullMQ Job.fromId() + ownership check
  → Returns { status, url? }

PDF Worker (separate PM2 process)
  → Picks up job from Redis queue
  → Deducts credit atomically (before render)
  → Renders React component → PDF buffer
  → Uploads to R2
  → Updates resume.pdfR2Key
  → Returns signed URL

The route handler returns in under 100ms. The client polls every 2 seconds. The worker does the heavy lifting independently. If the worker crashes mid-render, BullMQ retries automatically.

The Credit Deduction Race

This is the most subtle part of the entire system.

Credits must be deducted atomically. The naive approach — check credits, then deduct — has a race condition: two concurrent download requests both check "does this user have credits?" Both see creditsUsed < creditsTotal. Both pass. Both deduct. The user gets two PDFs for the price of one.

ScoreResume deducts credits with updateMany and a guard:

// lib/queue/pdf-worker.ts
const deducted = await prisma.pack.updateMany({
  where: {
    id: packId,
    creditsUsed: { lt: pack.creditsTotal },  // guard: only if still has credits
  },
  data: { creditsUsed: { increment: 1 } },
})
if (deducted.count === 0) {
  throw new Error("Pack credits exhausted — cannot generate PDF")
}

The WHERE creditsUsed < creditsTotal guard makes the check-and-deduct atomic. PostgreSQL evaluates it inside the UPDATE, holding a row lock. If two workers race, one wins and the other's deducted.count === 0 — it throws, the job fails, and BullMQ retries (by which point the user has no credits left).

The Idempotency Key — Preventing Double Charge on Retry

BullMQ retries failed jobs automatically. But what if the worker deducted a credit successfully, then crashed during the render before it could set processed = true? On retry, the worker deducts again. The user is charged twice for one PDF.

The fix is a Redis idempotency key:

// lib/queue/pdf-worker.ts
const deductKey = `pdf:deduct:${job.id}`
const acquired = await redis.set(deductKey, "1", "EX", 24 * 3600, "NX")

if (acquired === "OK") {
  // First attempt — deduct credit
  const deducted = await prisma.pack.updateMany(…)
  if (deducted.count === 0) {
    await redis.del(deductKey)  // allow future retries to re-check
    throw new Error("Credits exhausted")
  }
} else {
  // Key exists — this is a retry, credit was already deducted
  logger.info({ jobId: job.id }, "Credit already deducted — skipping")
}

SET key value EX 86400 NX is atomic. Only the first attempt gets "OK". Every retry gets null — "already deducted, skip." The key lasts 24 hours (matching removeOnFail: { age: 86400 }) so retries within the 24-hour window always see the correct state.

The Versioned R2 Key

Every PDF generation produces a new R2 path:

// lib/r2/keys.ts
export function resumePdfKey(userId: string, resumeId: string, version: string): string {
  return `resumes/${userId}/${resumeId}/${version}.pdf`
}

// In the worker:
const version = `${Date.now()}-${job.id}`
const r2Key = resumePdfKey(userId, resumeId, version)

Why versioned? The old pattern used a stable key (resumes/{userId}/{resumeId}.pdf). After the user edited their resume and re-downloaded, the browser, Cloudflare CDN, and R2's own edge all had the old bytes cached under the same URL. The new version appeared to work (200 response) but delivered stale content. A bug invisible in testing and infuriating for users.

Versioned keys eliminate the cache problem entirely. Each generation is a new, globally unique URL. Old versions become orphans — the worker deletes the previous pdfR2Key best-effort after updating the DB:

if (previousR2Key && previousR2Key !== r2Key) {
  deleteObject(previousR2Key).catch(e =>
    logger.warn({ e, previousR2Key }, "Failed to delete previous PDF — orphan left behind")
  )
}

The delete is fire-and-forget. A failed delete leaves an orphan R2 object (small cost), but never leaves the DB pointing at a deleted object (which would break the user's download).

The Job Deduplication Matrix

PDF jobs use a deterministic jobId = "pdf:final:{resumeId}". BullMQ treats this as a unique key — adding a job with an existing ID in waiting/active state is a no-op. This prevents double-clicks during render from creating two concurrent jobs.

But there is a subtlety: BullMQ removes completed jobs after 1 hour (removeOnComplete: { age: 3600 }). The Redis idempotency key lives 24 hours. If a user edits their resume and re-downloads within 1–24 hours of their last successful download, the completed job is gone from BullMQ but the idempotency key is still in Redis. A new job is added — same jobId. The worker runs. The idempotency key says "already deducted" — the user gets a free PDF.

The fix runs before adding a new job:

// lib/queue/pdf-queue.ts — addPdfJob for final jobs
const existing = await Job.fromId(getPdfQueue(), jobId)
if (existing) {
  const state = await existing.getState()
  if (state === "completed" || state === "failed") {
    await redis.del(`pdf:deduct:${jobId}`)  // clear stale key FIRST
    await existing.remove()                  // then remove old job
  }
  // waiting/active/delayed → dedup, return existing job
} else {
  // No job in queue — clear any stale key from the 1h-24h gap
  await redis.del(`pdf:deduct:${jobId}`)
}

The order matters: delete the Redis key before removing the BullMQ job. If the Redis delete fails, abort — the queue is intact, user retries. Reversed order: delete the job, Redis delete fails, new job is added with stale key → free PDF.

The 4-Minute Timeout

// lib/queue/pdf-worker.ts
const JOB_TIMEOUT_MS = 4 * 60 * 1000

async function processPdfJobWithTimeout(job) {
  const timeout = new Promise<never>((_, reject) =>
    setTimeout(() => reject(new Error("PDF job timed out")), JOB_TIMEOUT_MS)
  )
  return Promise.race([processPdfJob(job), timeout])
}

Without a timeout, a hung Puppeteer process or a network stall to R2 can block the worker indefinitely. concurrency: 1 means no other PDFs process while it's blocked. Users queue up. The timeout kills the job after 4 minutes, BullMQ retries, and the next attempt usually succeeds.

The Mental Model

PDF generation is a three-step transaction: deduct credit → render → upload. Each step can fail independently. The credit must be deducted first (to prevent races), the render must succeed before upload (or the credit is refunded), and the upload must succeed before the DB is updated (or the DB would point at a nonexistent object). The idempotency key prevents double-deduction across retries. Versioned R2 keys prevent stale CDN serving. The job deduplication matrix maps every possible queue state to the correct action.

If this were 10x scale

With 10 concurrent PDF downloads, concurrency: 1 would queue all jobs. Render time (3–8s) × 10 = 30–80s wait for the last user. Fix: increase concurrency, add a second worker process, or upgrade to a multi-core render pool. The current single-worker design is correct for early-stage; the BullMQ architecture makes horizontal scaling straightforward.

Chapter 6: The Scoring Engine — Deterministic by Design

Why No AI in Scoring

The Problem First

Resume scoring must be fair and explainable. If the same resume scores 68 one day and 71 the next because the AI "felt differently," users will notice. If you cannot explain exactly why a bullet lost 4 points, the score is useless as a tool for improvement.

The Insight

A scoring engine that changes output based on internal AI state is not a scoring engine — it is a black box with a number attached. Determinism is not a technical constraint here; it is a product requirement. Users must be able to trust that the score reflects the resume, not the weather.

The 5-Phase Pipeline

The engine in lib/scoring/engine.ts runs exactly five phases in order:

Phase 1: Section Awards
  → Award points for presence and quality of each section
  → Experience (scaled 0.4x/0.7x/1x based on entry count)
  → Skills (scaled by count: 0-3, 4-6, 7-9, 10+)
  → Summary (full if ≥50 chars, half otherwise)
  → Education, Projects, Certifications, Links, Format

Phase 2: Cap Logic
  → Cap score at 50 if skills = 0
  → Cap score at 40 if experience is empty (standard mode)
  → Prevent any single section from over-inflating the total

Phase 3: Flat Deductions
  → Weak openers: "responsible for", "helped with" → -4 each
  → Banned summary words: "hardworking", "team player" → -5 each (max 3)
  → Unquantified bullets: no number, no % → -4 each (max 5)
  → Repeated action verb (3+ times) → -5
  → Missing GitHub for tech roles → -10
  → Missing certifications when required → -10

Phase 4: Country + Role Checks
  → Photo policy: photo on US/UK resume → -15
  → Photo policy: no photo on Gulf resume → -10
  → Missing nationality (Gulf region) → -5
  → Missing visa status (Gulf region) → -5

Phase 5: Clamp
  → score = Math.min(100, Math.max(0, score))

Every phase adds to a positives[] and deductions[] array. The final breakdown is every rule that fired, with a reason and suggestion. The user sees exactly what cost them points and exactly what to fix.

Role and Country Configuration

The engine is parameterized. Each of 21 roles has a weight object:

// lib/scoring/role-configs.ts (example)
"react-developer": {
  weights: {
    experience: 30,
    skills: 20,
    summary: 10,
    education: 10,
    projects: 15,
    certifications: 5,
    links: 5,
    format: 5,
  },
  requiresGithub: true,
  requiresProjects: true,
  requiresCertifications: false,
}

A nurse role weights certifications heavily and does not require GitHub. A React developer role weights projects and requires GitHub. The same resume scores differently against different roles — correctly, because a React developer's resume evaluated for a nursing role is not a great nursing resume.

Country configs control region-specific requirements: photo policy, nationality, visa status, length norms.

The AI-Scoring Hybrid

The engine accepts an optional ResumeAnalysis from a Gemini pre-analysis. When available, it uses AI-detected bullet quality (is this bullet quantified? is it generic?). When absent, it falls back to regex detection. The scoring result records which version ran:

scoreVersion: analysis ? "2.0" : "1.0"

This means the determinism guarantee holds at each version: Version 1.0 is purely regex-based (fully deterministic). Version 2.0 uses AI analysis as input, but the scoring logic itself is still deterministic — the same AI output always produces the same score.

The Anti-Pattern Detectors

lib/scoring/anti-patterns.ts contains the detectors:

• detectWeakOpener(bullet) — regex against a list of weak openers: "responsible for", "helped", "worked on", "assisted with"
• detectBannedSummaryWords(summary) — matches overused resume clichés: "hardworking", "team player", "go-getter", "passionate"
• isUnquantified(bullet) — checks for absence of numbers, percentages, or scale indicators
• detectRepeatedVerb(bullets[]) — counts action verb occurrences; flags if any appear 3+ times

These are intentionally simple. A regex that misses a weak opener occasionally is fine — the engine will catch it on the next save. A regex that crashes is not fine.

The Test Suite

12 deterministic tests in lib/scoring/engine.test.ts. Run after every engine change:

npm run test:scoring

A broken rule fails silently — wrong score, no error, no indication. The only protection is the test suite. Each test asserts the exact score for a known resume/role/country combination. If Phase 3 inadvertently stops deducting for weak openers, test 7 fails immediately.

The Mental Model

A scoring engine is a pure function: (resumeData, roleSlug, countryCode) → ScoreResult. Pure functions are testable, auditable, and explainable. The moment you introduce state, randomness, or external calls into the scoring path, you lose all three properties. The AI analysis is pre-computed and passed as input — the engine treats it as data, not as a live dependency.

Chapter 7: The Data Layer

Schema Decisions — Why the Tables Are Shaped This Way

The Pack and its paymentId field

model Pack {
  id           String   @id @default(uuid())
  paymentId    String   @unique        // "pending_<packId>" until activated
  creditsTotal Int
  creditsUsed  Int      @default(0)
  expiresAt    DateTime?              // null until activation
  couponId     String?
  // ...
}

paymentId @unique is a database-level guard against two packs being activated with the same payment. Even if the application layer has a bug that calls activatePack twice, the unique constraint fires and prevents the second record. The pending_ prefix is a convention that encodes state without a separate column — queryable, self-documenting, and schema-efficient.

The WebhookEvent table

model WebhookEvent {
  id        String   @id    // Dodo payment ID — no uuid default; the gateway IS the key
  gateway   String
  eventType String
  payload   Json
  processed Boolean  @default(false)
  createdAt DateTime @default(now())
}

The primary key is the payment provider's ID. This is not an accident. By making the payment ID the PK, the unique constraint is automatic and the lookup is by primary key (fastest possible). ON CONFLICT on a PK is handled atomically by PostgreSQL. Two concurrent inserts with the same id — only one succeeds.

processed starts as false. It becomes true only after activation and coupon increment succeed, inside the same transaction. This means: if the handler crashes between the webhook event insert and the end of the transaction, processed remains false, and the retry re-runs activation (which is idempotent).

The Resume's pdfR2Key field

model Resume {
  pdfR2Key  String?   // null until first successful generation
  // ...
}

pdfR2Key is null for new resumes and for resumes that have been edited since their last PDF generation. The shouldInvalidatePdf() function in lib/resumes/resume-service.ts sets it to null on any edit that changes the visual output (format, content, colors). This null-check drives the fast path in /api/pdf/generate: if pdfR2Key is non-null, serve the cached R2 object. If null, queue a new job.

Prisma 7 and the Config Split

Prisma 7 moves the database URL out of schema.prisma and into prisma.config.ts:

// prisma.config.ts
import { defineConfig } from "prisma/config"
export default defineConfig({
  datasource: { url: process.env.DATABASE_URL },
})

schema.prisma has only provider = "postgresql" in the datasource block. This separation means the schema file can be committed to version control without fear of accidentally exposing environment variable patterns. It also means the config is TypeScript — autocomplete, type safety, no string misspellings.

The AiUsage Table

model AiUsage {
  id        String   @id @default(uuid())
  userId    String
  feature   String
  model     String
  inputTokens   Int
  outputTokens  Int
  costUsd   Decimal
  createdAt DateTime @default(now())
}

Every Gemini call is logged. This table serves two purposes: the per-user daily budget check (aggregate costUsd WHERE userId = X AND createdAt >= startOfDay) and the platform-wide budget check (aggregate all users). The daily budget is cheap to query because the window is narrow (same day) and indexed by createdAt.

The Mental Model

Schema design is about encoding invariants in the database itself, not just in the application layer. The paymentId @unique constraint prevents double-activation even if the application has a bug. The WebhookEvent primary key guarantees idempotent processing even under concurrent retries. The pdfR2Key null-check drives the caching logic. Each database-level constraint is a rule that fires even when the application layer fails.

Chapter 8: API Design

The Contract Between Frontend and Backend

The Mandatory Route Pattern

Every API route in ScoreResume follows exactly one pattern:

requireUser() → withRateLimit() → safeParse() → logic → response

This is documented in CLAUDE.md as non-negotiable. The order matters:

1. requireUser() first — auth before anything. An unauthenticated request never touches the rate limiter or the parser.
2. withRateLimit() second — rate limit by authenticated user ID, not by IP. A corporate NAT has dozens of users behind one IP; rate limiting by IP would block them all when one is abusive.
3. safeParse() third — only parse and validate input from authenticated, non-rate-limited callers. Parsing Zod on unauthenticated requests wastes CPU.
4. Logic last — business logic runs only on authenticated, rate-limited, validated input.

The Response Contract

Every route uses helpers from lib/api/response.ts:

ok(data)      → { data: T }           HTTP 200
err(message)  → { error: string }     HTTP 400
ApiErrors.UNAUTHORISED()  → 401
ApiErrors.TIER_REQUIRED() → 402
ApiErrors.FORBIDDEN()     → 403
ApiErrors.NOT_FOUND()     → 404
ApiErrors.RATE_LIMITED()  → 429
ApiErrors.INTERNAL()      → 500

The frontend always reads result.data or result.error. The HTTP status code drives routing logic on the client (402 = show upgrade modal, 429 = show retry timer). A consistent contract means every new feature can be integrated on the frontend without reading the source of the route.

Server-Only Fields

These fields never appear in user-submitted Zod schemas: isPaid, packId, score, pdfR2Key, scoreBreakdown. They are written exclusively by the scoring engine, PDF worker, and webhook handler:

// Correct — user cannot promote their own resume
const schema = z.object({
  title: z.string().optional(),
  targetRole: z.string().optional(),
  // isPaid is NOT here
})

If isPaid appeared in a PATCH schema for /api/resume/[id], a user could POST { isPaid: true } and get the paid experience for free. The protection is not validation — it is absence. A field that isn't in the schema cannot be written.

The Rate Limit Key Registry

Rate limit keys are pre-registered in lib/redis/rate-limit-configs.ts:

export const RATE_LIMITS = {
  AI_BULLET:       { limit: 40,  windowMs: 24 * 60 * 60 * 1000 },
  AI_SUMMARY:      { limit: 20,  windowMs: 24 * 60 * 60 * 1000 },
  AI_COVER_LETTER: { limit: 15,  windowMs: 24 * 60 * 60 * 1000 },
  WEBHOOK_DODO:    { limit: 60,  windowMs: 60 * 1000 },
  // ...
}

Using an unregistered key throws at runtime. The registry also makes the full rate-limit policy auditable in one file. When the bullet limit needs to change from 40 to 20, one file, one change, all routes updated.

The Sliding Window Rate Limiter

The rate limiter uses a Redis sorted set, not a counter. Each request is stored as a timestamped member. Old entries are removed before counting:

const pipeline = redis.pipeline()
pipeline.zremrangebyscore(key, '-inf', windowStart)   // remove expired
pipeline.zcard(key)                                    // count remaining
pipeline.zadd(key, now, `${now}-${Math.random()}`)    // add this request
pipeline.pexpire(key, windowMs + 1000)                // TTL
await pipeline.exec()

The random suffix on the member prevents collision (two requests at the same millisecond would overwrite each other without it). The pipeline executes all four commands atomically in one Redis round-trip. The windowMs + 1000 TTL adds a 1-second buffer for Redis's approximate TTL scheduling.

The Mental Model

An API contract is more valuable than any individual feature. Consistent response shapes, ordered middleware, and server-only field protection are rules that pay dividends across every future route. The cost of enforcing them is low (a code review check). The cost of not enforcing them compounds: inconsistent responses, security holes, and frontend code that special-cases every route.

Chapter 9: Failure and Resilience

What Happens When Things Break

The Redis Fallback

Redis is used for rate limiting and AI concurrency locks. If Redis goes down, the naïve options are: (a) fail open — allow all requests, zero rate limiting; (b) fail closed — reject all requests with 500.

ScoreResume uses a third option: an in-memory fallback:

// lib/redis/with-rate-limit.ts
try {
  return await checkRedisRateLimit(key, config)
} catch {
  logger.warn("Redis unavailable — falling back to in-memory rate limit")
  return checkMemoryRateLimit(key, config)
}

The in-memory fallback is per-process (not distributed), so it does not prevent abuse across PM2 cluster instances. But it prevents single-instance abuse and keeps the site up during a Redis outage. Imperfect protection beats a site-wide outage.

The Worker Shutdown Timeout

// worker.ts
async function shutdown() {
  const timeout = new Promise<never>((_, reject) =>
    setTimeout(() => reject(new Error("Shutdown timeout")), 30_000)
  )
  try {
    await Promise.race([worker.close(), timeout])
    logger.info("PDF worker shut down cleanly")
  } catch {
    logger.warn("PDF worker shutdown timed out — forcing exit")
  }
  process.exit(0)
}

process.on("SIGTERM", shutdown)
process.on("SIGINT", shutdown)

On deployment or shutdown, the worker drains active PDF jobs for up to 30 seconds before exiting. Without this, a SIGTERM during render would kill the worker mid-job. The credit was deducted, the render never completed, the job is stuck. With the grace period, in-progress renders have 30 seconds to complete.

The BullMQ Stall Detection

// lib/queue/pdf-worker.ts
const worker = new Worker("pdf-generation", handler, {
  lockDuration: 5 * 60 * 1000,    // 5 min — job is "locked" to this worker
  stalledInterval: 5 * 60 * 1000, // check for stalled jobs every 5 min
  concurrency: 1,
})

If the worker process is killed hard (OOM, SIGKILL) while processing a job, the job becomes "stalled" — no heartbeat, no completion. BullMQ's stall detection re-queues it after stalledInterval. The idempotency key (pdf:deduct:{jobId}) ensures the retry does not double-charge.

The Health Check

GET /api/health checks PostgreSQL and Redis, returns their status. UptimeRobot pings it every 5 minutes. A non-200 response triggers SMS + email alert within 10 minutes.

The Refund on Render Failure

// lib/queue/pdf-worker.ts
} catch (renderErr) {
  // Refund the credit — render failed, user shouldn't be charged
  if (packId) {
    await redis.del(`pdf:deduct:${job.id}`).catch(() => {})
    await prisma.pack.update({
      where: { id: packId },
      data: { creditsUsed: { decrement: 1 } },
    }).catch(e => logger.error({ e }, "Credit refund failed"))
  }
  throw renderErr
}

The deduction happens before the render. If the render fails, the credit is refunded before re-throwing. The Redis key is also deleted so the next BullMQ retry can deduct fresh (the render might succeed on retry with different conditions).

The Mental Model

Resilience is not about preventing failures — it is about defining what happens when they occur. For Redis: the site stays up with in-memory rate limiting. For worker crashes: BullMQ re-queues, idempotency prevents double-charge. For render failures: the credit is refunded. For deployment: the 30-second drain ensures in-progress jobs complete. Each failure mode has a defined behavior.

Chapter 10: The Gaps

A Brutally Honest Assessment of What Is Missing

Gap 1: No Point-in-Time Recovery

What is missing: Daily pg_dump backups to Cloudflare R2. No WAL archiving.

What breaks: A bug corrupts the resumes table. You discover it 6 hours later. You can restore to yesterday's backup — but lose 6 hours of user resumes. WAL archiving would let you restore to any moment: 5 minutes before the bug triggered.

Simplest fix: Configure wal_level = replica on PostgreSQL and archive WAL segments to R2 using pgBackRest or wal-g. Enables point-in-time recovery to any second. Cost: minimal storage ($0.015/GB on R2), significant operational complexity.

Gap 2: No Circuit Breaker on Gemini

What is missing: If Gemini has an extended outage (not just a 429 but a sustained 503), every AI request retries 3 times before giving up. Under load, this creates a retry storm: 100 users × 3 retries × 30-second timeouts = 300 long-running requests blocking event loop workers.

What breaks: The Next.js server becomes unresponsive to non-AI requests during a Gemini outage. Rate limiting and scoring (which don't use Gemini) become unreachable.

Simplest fix: Add the opossum circuit breaker library around callGemini(). Configure it to open after 50% failure rate in a 10-second window. Open circuit returns 503 immediately without hitting Gemini.

Gap 3: Orphan Pack Cleanup Is Per-Process

What is missing: cleanupOrphanPacks() uses a module-level _lastCleanup timestamp to throttle — at most once per 30 minutes per process. With PM2 cluster mode (2 processes), cleanup runs up to twice per 30 minutes — once per process. This is harmless redundancy, not a bug.

What actually breaks: The throttle is reset on process restart. If the process restarts frequently (deployment, crashes), the throttle resets and cleanup runs on every restart — potentially more than once per 30 minutes.

Simplest fix: Move the throttle to Redis: redis.set("cleanup:orphan_packs:last", Date.now(), "EX", 1800, "NX"). Only one process runs cleanup within 30 minutes, regardless of restarts.

Gap 4: Resume Data Is an Opaque JSON Column

What is missing: Resume.data is a Json column. Prisma returns it as unknown. Every access requires a cast: resume.data as unknown as ResumeData. There is no database-level schema validation on resume data.

What breaks: A future schema migration changes a field name in ResumeData. Old resumes in the database still have the old field name. The TypeScript types say the field exists, but the data doesn't. The scoring engine silently scores 0 for the missing section.

Simplest fix: A migration that validates existing JSON rows against the current ResumeData schema. A long-term fix is versioning resume data with a dataVersion column and a migration script.

Gap 5: No Cache Stampede Prevention

What is missing: Multiple analytics and pricing endpoints have no Redis caching. Under sudden load (a feature launch, a viral post), many users hit the same uncached endpoint simultaneously. Each request goes to PostgreSQL. The DB query time spikes.

Simplest fix: Cache pricing promo responses in Redis with a 60-second TTL. Add TTL jitter (Math.random() * 10 seconds) so all entries don't expire simultaneously.

Epilogue: The Whiteboard Test

This book set out to teach systems thinking through reverse-engineering. Every chapter started from a problem, exposed the naive failure, arrived at the insight, and showed the real code as proof.

Here is the whiteboard test: given a blank whiteboard and these requirements — resume scoring SaaS, PDF downloads, AI writing features, payment packs — can you now sketch the architecture and justify every decision?

Start with the data. There are resumes (user-owned JSON), packs (credit bundles with expiry), and PDF jobs (async rendering tasks). The pack's paymentId encodes state — pending vs active. The resume's pdfR2Key drives the download fast path — cached vs re-generate.

Add the request path. Browser → CDN (IP geolocation for currency) → Nginx → Next.js → PostgreSQL via Prisma. Auth before rate limiting. Rate limiting before parsing. Parsing before business logic. Never in any other order.

Add the async path. PDF generation is slow and CPU-intensive — it belongs in a BullMQ queue, not a request handler. The worker deducts credits before rendering (not after), refunds on failure, uses an idempotency key to prevent double-charge on retry. Versioned R2 keys eliminate CDN staleness.

Add the payment path. Dodo fires webhooks that may arrive multiple times. The WebhookEvent table deduplicates by payment ID. activatePack is idempotent. Both layers together make double-activation structurally impossible. Atomic transaction wraps activation + coupon increment + webhook marking.

Add the AI path. Three independent limiting layers: tier gate (entitlement), rate limit (temporal throttle), budget cap (financial circuit breaker). A per-user Redis mutex prevents concurrent requests from racing past the budget check before any writes. Gemini output is stripped of markdown fences before JSON parsing.

Add the scoring path. A pure function: (resumeData, role, country) → ScoreResult. Deterministic. No external calls in the scoring path itself. AI analysis is pre-computed and passed as input. 12 tests that fail immediately if any rule breaks.

If you can draw this on a whiteboard and explain every choice when challenged — why pending_ prefix instead of a status column, why credit deduction before render, why the Redis key must be deleted before removing the BullMQ job, why CF-IPCountry instead of body country code — you are thinking like a systems engineer.

That is the goal. Not knowing what ScoreResume does. Knowing why.

The source of truth for this book is the ScoreResume codebase itself. All code snippets are from the actual implementation.