ScoreResume: Interview Preparation Guide

Real questions, grounded in real code. Four depth layers per question: WHAT / WHY / HOW DEEP / STRESS.

How to Use This Book

Every question has four layers:

• WHAT: The surface answer. Enough for a 30-second explanation.
• WHY: The reasoning behind the decision. Why this, not something else?
• HOW DEEP: The implementation detail. Show you've read the code.
• STRESS: What breaks under pressure, edge cases, failure modes.

Read all four layers before your interview. In the room, start with WHAT. If the interviewer says "go deeper," move to WHY. Only go to HOW DEEP if they push again. STRESS answers signal senior-level thinking — volunteer them to signal depth.

Section 1: Architecture

Q1: Walk me through ScoreResume's overall architecture.

WHAT: ScoreResume is a Next.js 14 App Router monolith. Single codebase, single VPS deployment. Browser requests hit Cloudflare (DDoS + CDN + IP geolocation), then Nginx (SSL + reverse proxy), then the Next.js process on port 3001. Heavy async work — PDF generation — runs in a separate BullMQ worker process. Files go to Cloudflare R2. Data is in PostgreSQL. Redis handles rate limiting, queue state, and concurrency locks.

WHY: Monolith was the right choice for a bootstrapped product. Microservices add operational overhead — separate deploys, service discovery, network hops between services. A monolith with clear module boundaries (lib/payments/, lib/scoring/, lib/ai/) gives modularity without the infrastructure cost. The PDF worker is the only separate process, because it is CPU-intensive and must be independently restartable without taking down the web server.

HOW DEEP: Two applications share one VPS: ScoreResume (port 3001) and AttemptIQ (port 3000). Nginx routes by domain. PM2 runs both as separate cluster processes. They share the VPS's RAM and CPU but have strictly separate databases, Redis key namespaces, and Nginx configs. The PDF worker runs as scoreresume-worker in PM2, separate from the Next.js cluster.

STRESS: What happens if the VPS goes down? Everything stops — web, worker, queue. There is no failover. This is a known bootstrapped-stage trade-off. The mitigation is monitoring (UptimeRobot, Netdata) and fast recovery via PM2 restart scripts. The next scaling step is a second VPS for the worker.

Q2: How does ScoreResume handle authentication?

WHAT: Better Auth manages sessions. On login, a random session token is stored in the sessions table and set as an HttpOnly, Secure, SameSite=Lax cookie. Every protected request validates the token against the database. Auth runs in the Node.js runtime inside route handlers — not in Edge middleware.

WHY: Edge middleware cannot run Prisma (no Node.js APIs). Doing real auth in middleware would require an HTTP round-trip back to the API on every page navigation. Two-layer approach: Edge middleware checks cookie presence (fast, no DB hit) and redirects obviously unauthenticated users. Real auth (token lookup, ban check) happens in requireUser() in the route handler.

HOW DEEP:

// middleware.ts — fast UX gate only
const sessionCookie = request.cookies.get(SESSION_COOKIE_NAME_SECURE) ??
                      request.cookies.get(SESSION_COOKIE_NAME)
if (!sessionCookie?.value) return NextResponse.redirect("/login")

// lib/api/auth-guard.ts — real auth
const { user } = await getSession()  // DB lookup
if (!user) throw "NOT_AUTHENTICATED"
const dbUser = await prisma.user.findUnique({ where: { id: user.id }, select: { isBanned: true } })
if (dbUser?.isBanned) throw "NOT_AUTHENTICATED"

requireUser() throws a string ("NOT_AUTHENTICATED"), not an Error object. This prevents Sentry from capturing a stack trace on every unauthenticated request — which would create noise and cost.

STRESS: What if the session table grows to millions of rows? The sessions table is indexed by token. The lookup is O(log n) by index. Better Auth handles session cleanup — expired sessions are purged. What if someone brute-forces session tokens? The token is random (crypto-secure entropy), not an incrementing ID. Guessing a valid token is statistically impossible. What about CSRF? SameSite=Lax prevents cross-origin POSTs from carrying the cookie. The cookie itself is the CSRF token.

Q3: How does the API enforce a consistent middleware ordering?

WHAT: Every API route follows the same pattern: requireUser() → withRateLimit() → safeParse() → logic. This ordering is documented in CLAUDE.md as non-negotiable.

WHY: The ordering prevents wasted work. Auth first: unauthenticated requests never touch the rate limiter or parser — no Redis or CPU wasted on attackers. Rate limit after auth: limits apply per authenticated user, not per IP. Parsing after rate limit: Zod validation is CPU work that shouldn't run for rate-limited requests. Business logic last: only runs on validated, authenticated, non-rate-limited input.

HOW DEEP: requireUser() throws a string to the global error handler when auth fails. The global error handler catches strings from requireUser and returns 401, catches AiBudgetExceededError and returns 503, catches everything else and returns 500. Routes don't need their own error handling for auth or budget — it's centralized.

STRESS: What if someone puts safeParse() before requireUser()? Unauthenticated requests now parse the body. A malformed body from an attacker causes a Zod parse error and a 400 response — slightly more information than a 401. Also wastes CPU. Not catastrophic, but wrong. What if withRateLimit() skips on Redis failure? The in-memory fallback kicks in. Single-instance rate limiting only — imperfect but keeps the site up.

Section 2: Payments

Q4: How does the Pack lifecycle work?

WHAT: A Pack starts with paymentId = "pending_<packId>". When payment is confirmed via webhook, activatePack() replaces the paymentId with the real Dodo payment ID and sets expiresAt. The paymentId field doubles as a state flag — no separate status column needed.

WHY: expiresAt is calculated at activation, not at order creation. If we set expiry when checkout begins, a user who pays 3 days later loses 3 days of their pack. Setting expiry at activation means "your time starts when you pay." The pending_ prefix encodes state without a schema migration — one field does two jobs.

HOW DEEP:

// lib/payments/pack-service.ts
export async function activatePack({ packId, paymentId, … }) {
  const pack = await prisma.pack.findUnique({ where: { id: packId } })
  if (!pack.paymentId.startsWith("pending_")) {
    logger.warn("Pack already active — skipping")
    return pack  // idempotent no-op
  }
  return prisma.pack.update({
    where: { id: packId },
    data: { paymentId, expiresAt: calcExpiryDate(pack.packType) }
  })
}

The startsWith("pending_") check makes activatePack idempotent — safe to call multiple times. If a webhook retry calls it twice, the second call sees a real paymentId and exits without doing anything.

STRESS: What if calcExpiryDate has a bug and returns a past date? The pack is activated but immediately expired. The user cannot use their credits. The fix is validation before the update: assert that expiresAt > new Date(). What if the pack ID in the webhook doesn't match any database record? findUnique returns null. The route returns 400 with a logged error. Dodo retries — same result. An alert fires. Manual investigation resolves it.

Q5: How does webhook idempotency work?

WHAT: Dodo retries webhooks that don't get a 200 response within 10 seconds. To prevent double-activation, ScoreResume uses a WebhookEvent table with the Dodo payment ID as primary key. The first delivery inserts a row; retries hit the unique constraint (P2002) and check the processed flag. If processed = true, return 200 immediately. If processed = false, the previous attempt crashed — retry activation.

WHY: The database unique constraint is the most reliable idempotency mechanism. It uses the same PostgreSQL transaction guarantees as the rest of the data. In-memory deduplication (a Set or Map) doesn't survive process restarts. Redis-based deduplication survives restarts but adds a dependency for a critical payment flow. The WebhookEvent table is auditable — you can see every webhook ever received, its payload, and whether it was processed.

HOW DEEP:

try {
  await prisma.webhookEvent.create({ data: { id: payment.payment_id, processed: false, … } })
} catch (e) {
  if (e.code === "P2002") {
    const existing = await prisma.webhookEvent.findUnique({
      where: { id: payment.payment_id }, select: { processed: true }
    })
    if (existing?.processed) return new Response("OK", { status: 200 })
    // else: previous attempt crashed mid-activation — fall through to retry
  } else throw e
}
// ... activate pack, then:
await prisma.webhookEvent.update({ where: { id: payment.payment_id }, data: { processed: true } })

The processed flag transitions from false to true inside the same transaction as activation. If the transaction rolls back, processed stays false — the retry will re-run activation correctly.

STRESS: What if two webhook deliveries arrive simultaneously — genuine concurrent delivery? Both hit the WebhookEvent.create() attempt simultaneously. Only one succeeds. The other gets P2002 and reads processed. At this exact moment, the first delivery hasn't finished activation yet — processed = false. The second delivery falls through and tries to activate. activatePack checks paymentId.startsWith("pending_"). If the first delivery already activated, this is a no-op. If not, both activations race. prisma.pack.update (not updateMany) on the same row — PostgreSQL serializes them. The second update overwrites the first with the same values. Safe.

Q6: How does ScoreResume detect a user's country for pricing?

WHAT: ScoreResume reads the CF-IPCountry header set by Cloudflare, not a country code from the request body. This header is based on real IP geolocation and cannot be spoofed by the client.

WHY: INR (India) pricing is cheaper than USD pricing. If the country came from the request body, a US user could POST { countryCode: "IN" } and pay ₹99 instead of $5.99 — a revenue exploit. Cloudflare sets CF-IPCountry based on the actual IP address of the connecting client. The client cannot forge it because Cloudflare rewrites it from its own geolocation database, not from the client's request.

HOW DEEP:

// app/api/payments/create-order/route.ts
const cfCountry = request.headers.get("CF-IPCountry") ?? countryCode  // fallback for local dev
const productId = getDodoProductId(planId, cfCountry)

In local development, Cloudflare is absent — CF-IPCountry doesn't exist. The body's countryCode is the fallback for development only. In production, the Cloudflare header always exists and always wins.

STRESS: What if Cloudflare is temporarily bypassed (someone accesses the VPS IP directly)? The Nginx config should block direct IP access — only Cloudflare's IP ranges should be allowed. If this is not configured, a user accessing the VPS IP directly gets no CF-IPCountry header, falls through to the body value, and can send any country code. This is a configuration gap — Nginx allow/deny rules should restrict access to Cloudflare IPs only.

Section 3: PDF Pipeline

Q7: Why is PDF generation in a queue instead of the request handler?

WHAT: PDF rendering takes 3–8 seconds and is CPU-intensive. Doing it synchronously in the request handler would block the HTTP connection for 8 seconds, hold a server thread, and make the server unresponsive to other requests during concurrent PDF requests. The queue decouples triggering (fast, < 100ms) from processing (slow, in a separate worker process).

WHY: A request handler should return as fast as possible. A 200ms handler can serve 5 requests/second on a single thread. An 8-second handler can only serve 0.125 requests/second — and blocks everything else during those 8 seconds. The queue pattern also gives retry, observability, and isolation: a PDF worker crash doesn't affect the web server.

HOW DEEP: BullMQ uses jobId = "pdf:final:{resumeId}" for deduplication. Two concurrent click-to-download attempts for the same resume reuse the same BullMQ job — only one render happens. The client polls GET /api/pdf/status/[jobId] every 2 seconds. The route checks BullMQ job state and returns a fresh signed R2 URL when state === "completed".

STRESS: What if the worker is down when jobs arrive? Jobs sit in the BullMQ waiting set in Redis. Redis persists data to disk (AOF or RDB). When the worker restarts, it picks up waiting jobs automatically. What if Redis goes down? The queue is lost. Jobs cannot be added. The web server's queue calls fail. This is a known single-point-of-failure at the current scale. Mitigation: Redis persistence enabled, UptimeRobot alert on queue failures.

Q8: How does ScoreResume prevent a user from being charged twice for one PDF?

WHAT: Two mechanisms: (1) atomic updateMany with a WHERE creditsUsed < creditsTotal guard prevents concurrent worker races. (2) A Redis idempotency key (pdf:deduct:{job.id}, NX, 24h TTL) prevents double-deduction on BullMQ retries.

WHY: Two different threats: race condition (concurrent workers, same job somehow duplicated) and retry (BullMQ retrying a failed job that already deducted). The atomic SQL guard prevents the race. The Redis key prevents the retry double-charge. They're independent — each covers a different failure mode.

HOW DEEP:

// Threat 1 — SQL atomicity
const deducted = await prisma.pack.updateMany({
  where: { id: packId, creditsUsed: { lt: pack.creditsTotal } },
  data: { creditsUsed: { increment: 1 } },
})
if (deducted.count === 0) throw new Error("Credits exhausted")

// Threat 2 — Idempotency on retry
const acquired = await redis.set(`pdf:deduct:${job.id}`, "1", "EX", 86400, "NX")
if (acquired !== "OK") {
  // Prior attempt deducted — skip deduction, still render
}

On render failure, the credit is refunded AND the Redis key is deleted. This allows the next BullMQ retry to deduct fresh — the render might succeed next time.

STRESS: What if the Redis key expires (24h) and BullMQ retries after 24h? This would be a 24-hour delay on a PDF job — far beyond BullMQ's configured retry window. The job would have failed permanently before the key expired. What if the worker deducts, then crashes before setting the Redis key? This is impossible — the SET NX is called before updateMany. The key is set first, deduction happens second. If the crash happens between the two calls, the key exists but the deduction didn't happen — the next retry sets NX (gets null — key exists), skips deduction, and... doesn't deduct. The user gets a free PDF in this narrow crash window. The fix would be to set the key and perform the DB deduction in a Lua script — overkill at current scale.

Q9: Why are PDF R2 keys versioned?

WHAT: R2 keys include a timestamp and job ID: resumes/{userId}/{resumeId}/{timestamp}-{jobId}.pdf. Each generation is a new, unique key. The previous key is deleted best-effort after the DB is updated.

WHY: Without versioning, a stable key (resumes/{userId}/{resumeId}.pdf) is cached by Cloudflare CDN, the browser, and R2's own edge network. After a user edits their resume and re-downloads, the URL is the same — the caches return the old bytes. The user sees their old resume. This is a cache staleness bug that's invisible in development (no CDN) and infuriating in production.

HOW DEEP:

const version = `${Date.now()}-${job.id}`
const r2Key = resumePdfKey(userId, resumeId, version)
// ... render and upload to r2Key
await prisma.resume.update({ data: { pdfR2Key: r2Key } })
// Clean up old key — fire-and-forget
if (previousR2Key) deleteObject(previousR2Key).catch(e => logger.warn(e))

The previous key deletion is best-effort. A failed delete leaves an orphan R2 object — minor storage cost (~$0.000005/month per PDF). Never deleting the previous R2Key first and updating the DB second — that would leave the DB pointing at a deleted object.

STRESS: What if the timestamp-based version has collisions? Date.now() has millisecond precision. Two jobs for the same resume within the same millisecond get the same timestamp component. But job.id is the BullMQ job ID — a UUID. No collision. What if R2 upload succeeds but DB update fails? The DB still has the old pdfR2Key. The new R2 object is an orphan — no DB record points to it. The worker's catch block re-throws, BullMQ retries, the next attempt renders and uploads again with a new version. The first orphan is never cleaned up automatically — a manual R2 audit would be needed to remove it.

Section 4: AI Features

Q10: How does ScoreResume prevent a single user from spending all the AI budget?

WHAT: Three independent limiting layers: (1) tier gate — entitlement check (must have the right pack type), (2) rate limit — sliding window per feature per user, (3) cost budget — per-user $2/day cap enforced in checkAiBudget(). Plus a per-user Redis mutex to prevent concurrent requests from racing the budget check.

WHY: One limit doesn't cover all attack surfaces. A tier gate alone doesn't stop a paid user from spamming requests. A rate limit alone doesn't prevent one expensive request. A budget cap alone has a TOCTOU race — five concurrent requests all pass the check before any usage is logged. The mutex closes the race.

HOW DEEP:

// lib/ai/cost-guard.ts
const lock = await redis.set(`ai:lock:${userId}`, "1", "EX", 60, "NX")
if (lock !== "OK") throw new AiBudgetExceededError("concurrent request in progress")
try {
  await checkSpend(userId)   // aggregate AiUsage for today
  // ... call Gemini
  await logAiUsage(userId, cost)
} finally {
  await redis.del(`ai:lock:${userId}`)
}

The mutex is per-user. User A and User B can have concurrent AI calls — only the same user is serialized. The 60-second TTL is a dead-man's switch: if the server dies mid-call without releasing the lock, it auto-expires in 60 seconds.

STRESS: What if Gemini returns a response that's 10x larger than expected? The budget check passed at $0.10 estimated cost. The actual cost is $1.00. The daily budget cap is exceeded after the call, not before. The next checkAiBudget() call returns 503. The over-spend of one call is bounded — it can't exceed the maximum single-call cost, which is bounded by the maximum response token limit configured in the callGemini() parameters.

Q11: Why does ScoreResume use two different Gemini models?

WHAT: Starter (free) users get gemini-2.5-flash-lite. Paid users get gemini-2.5-flash. Flash is ~5x more expensive but produces noticeably better output for complex technical writing.

WHY: Two reasons: cost and quality. Free users incur real cost even with the lite model. Running Flash for all users would make the free tier financially unsustainable. Paid users have a reasonable expectation of better service — the model tier is part of the value proposition.

HOW DEEP: The model selection happens at the call site:

const model = pack?.packType !== "starter"
  ? GEMINI_MODEL_PRO    // "gemini-2.5-flash"
  : GEMINI_MODEL_LITE   // "gemini-2.5-flash-lite"

COSTS is a map from model string to { input: $/M tokens, output: $/M tokens }:

const COSTS = {
  "gemini-2.5-flash-lite": { input: 0.10, output: 0.40 },
  "gemini-2.5-flash":      { input: 0.30, output: 2.50 },
}

The output cost disparity (0.40 vs 2.50 per million tokens) means heavy users on Flash cost 6x more in output tokens alone. For a user generating 40 bullet rewrites/day, the daily AI cost difference is significant.

STRESS: What if Google deprecates gemini-2.5-flash-lite? The constant GEMINI_MODEL_LITE is defined once — update one string, all routes pick it up. What if the Flash model is unavailable (Google outage on Pro tier only)? The error propagates as a Gemini API error — the route returns 503, user retries. No automatic fallback to Flash Lite for paid users — a deliberate choice to not silently degrade paid service quality.

Section 5: Scoring Engine

Q12: How does the scoring engine work, and why is it deterministic?

WHAT: Five phases run in sequence: (1) Section Awards — points for presence and quality of each section. (2) Cap Logic — caps score if critical sections are empty. (3) Flat Deductions — points off for anti-patterns: weak openers, banned words, unquantified bullets, repeated verbs. (4) Country/Role Checks — region-specific rules (photo policy, nationality). (5) Clamp — Math.min(100, Math.max(0, Math.round(score))). Same input always produces the same output.

WHY: Determinism is a product requirement. If the score changes between saves without the resume changing, users notice and lose trust. Non-determinism also makes testing impossible — you can't assert score === 73 if the score floats based on AI state. The engine is a pure function: (resumeData, role, country) → ScoreResult.

HOW DEEP: The engine accepts an optional ResumeAnalysis from Gemini. When provided, the analysis is pre-computed data — passed in, not fetched. The engine treats it as a deterministic input. Both paths (with and without analysis) are tested with exact integer assertions:

test("react-dev resume scores 73", async () => {
  const result = await scoreResume(resume, "react-developer", "US")
  assert.strictEqual(result.score, 73)  // not "approximately 73"
})

npm run test:scoring must run after every engine change. Wrong scores fail silently — a broken scoring rule returns 200 with a wrong number.

STRESS: What if role weights don't sum to 100? The sum doesn't need to be 100 — the weights are relative, not absolute. The max possible score is the sum of all weight values for a perfect resume. Scores above 100 are clamped. What if a new role is added without a config? The engine falls back to a default weight distribution. The fallback should be the same as the most general role ("software-engineer") — this is not currently enforced in the code and is a gap.

Q13: How does the scoring engine handle country-specific rules?

WHAT: Phase 4 checks region-specific conventions: photo on a US/UK resume deducts 15 points, no photo on a Gulf resume deducts 10 points, missing nationality on a Gulf resume deducts 5 points. Country codes come from the caller, not detected within the engine.

WHY: Resume conventions vary significantly by region. In the US and UK, a photo on a resume signals unfamiliarity with local norms — it raises concerns about age/race discrimination liability and looks unprofessional. In Gulf countries (UAE, Saudi Arabia, etc.), a photo is expected — its absence is the abnormality. Hardcoding these rules in the engine makes them testable and auditable.

HOW DEEP:

// lib/scoring/engine.ts — Phase 4
const gulfCountries = new Set(["AE", "SA", "QA", "KW", "BH", "OM"])
const isGulf = gulfCountries.has(countryCode)
const isAngloSaxon = ["US", "GB", "CA", "AU"].includes(countryCode)

if (isAngloSaxon && resumeData.hasPhoto) {
  deductions.push({ reason: "Photo on resume", points: -15, suggestion: "Remove photo for US/UK applications" })
}
if (isGulf && !resumeData.hasPhoto) {
  deductions.push({ reason: "Missing photo", points: -10, suggestion: "Add a professional photo for Gulf applications" })
}

The countryCode is provided by the caller — the user selects their target country in the UI. The engine does not detect country from IP or any external source.

STRESS: What if the user selects the wrong country? The engine scores correctly for the selected country, which may not match reality. This is user error, not engine error. The UI provides country selection — the engine trusts the input. What if new country-specific rules are needed? Add a case to Phase 4, add a test, run npm run test:scoring. The engine doesn't need schema changes — country configs are code.

Section 6: Rate Limiting

Q14: How does the sliding window rate limiter work?

WHAT: Each user's requests are stored in a Redis sorted set, keyed by user ID and feature. The member is {timestamp}-{random}, the score is the timestamp. Before each request, entries older than windowMs are removed. The count of remaining entries is the number of requests in the current window. If count >= limit, reject.

WHY: A fixed-window counter resets at exact clock boundaries. This allows "bursting" — 60 requests in the last second of minute 1 plus 60 requests in the first second of minute 2 = 120 requests in 2 seconds. The sliding window has no boundary — every request looks back exactly windowMs milliseconds from now. Bursting against the boundary is impossible.

HOW DEEP: Four commands in a single pipeline:

ZREMRANGEBYSCORE key -inf (now - windowMs)   → remove expired entries
ZCARD key                                      → count remaining
ZADD key now member                            → add this request
PEXPIRE key (windowMs + 1000)                 → reset TTL

results[1] (the ZCARD result) is the count before adding this request. If results[1] >= limit, this request would be the (limit+1)th — reject it. The +1000ms TTL buffer prevents the key from expiring while its youngest member is still in the window.

STRESS: What if two requests arrive at exactly the same millisecond? They would have the same timestamp. Without the random suffix, the second ZADD would update the score of the existing member (sorted sets require unique members). The count would show 1 instead of 2 — both requests would be allowed when only one should be. The random suffix makes this collision statistically impossible. What if Redis is down? The fallback is an in-memory rate limiter using a Map of timestamp arrays. It's per-process (not distributed), but it prevents single-instance abuse during a Redis outage.

Section 7: Infrastructure

Q15: Why does ScoreResume use PgBouncer?

WHAT: PgBouncer is a connection pooler that sits between the application and PostgreSQL. It multiplexes many application connections onto fewer PostgreSQL server connections. ScoreResume uses it in transaction mode — a server connection is borrowed only while a transaction is active.

WHY: PostgreSQL has a finite max_connections limit. Without PgBouncer, each PM2 cluster instance holds up to 10 Prisma connections open — idle but held. Four instances × 10 connections × 2 applications = 80 PostgreSQL connections. Under load, this saturated the connection limit and caused 502 errors (Prisma couldn't acquire a connection). PgBouncer reduces the actual PostgreSQL server connections to a small fixed pool.

HOW DEEP: A load test (50 VUs on k6) before PgBouncer produced 630 × 502 errors over a 5-minute test. After adding PgBouncer:

Application connections (PM2): up to 80
PgBouncer server connections: 20 (10 per application)
PostgreSQL max_connections used: 20

In transaction mode, a server connection is returned to the pool at COMMIT/ROLLBACK. Prisma's idle connections don't hold PostgreSQL connections — they hold PgBouncer connections, which are pooled and cheap.

STRESS: PgBouncer's transaction mode is incompatible with SET commands outside transactions, prepared statements across sessions, and advisory locks. Prisma's $queryRaw with SET for session variables would fail silently. The mitigation is to always use session mode for any connection that needs session-level state — but this defeats the purpose of the pool.

Q16: How does PM2 deployment work on the VPS?

WHAT: PM2 manages the Next.js cluster and the BullMQ worker as two separate processes. Deployment uses a specific sequence: pm2 stop X && pm2 delete X && pm2 start ecosystem.config.js && pm2 save. Never pm2 restart.

WHY: pm2 restart performs a rolling restart but may leave stale process state — environment variables from the old config are not re-read, the old module cache is not cleared. stop → delete → start creates a clean process from scratch, reading the latest ecosystem.config.js and all environment variables fresh.

HOW DEEP: The PDF worker has a 30-second graceful shutdown:

process.on("SIGTERM", async () => {
  await Promise.race([worker.close(), timeout30s])
  process.exit(0)
})

When PM2 stops the worker, it sends SIGTERM. The worker drains active PDF jobs for up to 30 seconds before exiting. Without this, a pm2 stop during render would kill mid-job — credit deducted, render never completed, user is stuck. The 30-second grace period gives in-progress renders time to complete.

STRESS: What if a deploy happens while 3 users are actively downloading PDFs? The web server stops accepting requests (Nginx starts queuing them or returning 502). The worker receives SIGTERM and drains for 30 seconds. If all three renders complete within 30 seconds, users get their PDFs — they may see a loading spinner but not an error. If a render takes longer than 30 seconds (unusual — typical is 3–8s), the worker force-exits. That job is retried by BullMQ. The idempotency key prevents double-charging on the retry.

Section 8: Failure Modes and Scaling

Q17: What happens if Gemini goes down for 2 hours?

WHAT: All AI features return 503. The retry logic (3 attempts, exponential backoff) fires for each request but eventually gives up. ScoreResume is available — scoring, PDF download, and all non-AI features work. The Gemini outage is isolated to AI features.

WHY: Gemini calls are in try/catch. A Gemini error is caught, logged, and returned as 503 to the client. The client shows "AI temporarily unavailable." Non-AI routes don't call Gemini — they're unaffected. The concurrency lock is released in a finally block — a Gemini timeout releases the lock even if the call never completed.

HOW DEEP: The current code has no circuit breaker. During a sustained Gemini outage with high user load, each request retries 3 times with up to 8-second delays. 100 concurrent AI requests × 3 retries × 8s max delay = potential long-tail requests tying up server threads. A circuit breaker (e.g., opossum library) would open after a failure threshold and return 503 immediately without retrying — reducing server load during an outage.

STRESS: What if Gemini comes back but is degraded — responding slowly with 30-second latency? The retry timeout is currently set by the Gemini SDK default (varies). A 30-second response still completes — the user waits 30 seconds. The concurrency lock is held for 30 seconds. Other users wanting AI features for the same user are blocked for 30 seconds (lock TTL is 60s). After the long response, the lock is released and usage is logged. This is functional but slow. The fix is an explicit timeout in callGemini() — abort the request after N seconds and retry.

Q18: How would you scale ScoreResume to 100x the current load?

WHAT: Current bottlenecks at 100x: single Next.js server process group, single BullMQ worker process, single PostgreSQL instance, single Redis instance. Each needs to scale independently based on actual bottleneck.

WHY: Scale the bottleneck, not everything. At 100x: PDF rendering CPU would saturate first (CPU-bound), then database connections (query throughput), then Redis (command throughput — Redis is fast enough for very high load), then network bandwidth (R2 uploads).

HOW DEEP:

PDF Pipeline (most likely first bottleneck):

• Increase worker concurrency from 1 to 4 (use all cores for render)
• Add a second VPS for additional worker instances
• BullMQ natively supports multiple workers pointing at the same Redis queue

Database:

• Read replicas for scoring and status-check queries (read-heavy, stale-ok)
• Increase PgBouncer pool size
• Add indexes on AiUsage.createdAt (budget queries) and Pack.userId + Pack.expiresAt (pack lookups)

Web Server:

• Move to multi-VPS with a load balancer in front
• Rate limiting must move from per-process in-memory fallback to always-Redis (already the primary path)
• Session storage moves from DB to Redis for hot path

AI (Gemini):

• Add a circuit breaker
• Add model-level caching for identical prompts (Redis key = hash of prompt, model)
• Consider a second Gemini project key for burst headroom

Scoring Engine:

• Already CPU-only, no external calls — scales with CPU cores
• Results can be cached per (resumeData hash, role, country) in Redis with a short TTL

STRESS: The hardest thing to scale is the PDF render step — it's CPU-intensive, requires the full React component tree to be loaded per render, and cannot be cached (output depends on current resume state). Moving render to a dedicated render farm (serverless functions or a dedicated render service) would isolate this bottleneck from the web server. This is the architectural change the current monolith defers.

Quick Reference: Key Numbers

Quick Reference: Key Invariants

These invariants hold at all times. Violating any of them is a production bug.

1. Auth before rate limit before parse before logic. Always. No exceptions.
2. checkAiBudget() before every Gemini call. Missing this is a financial hole.
3. Credit deduction before PDF render. Deduct first, refund on failure, never deduct after.
4. Redis key deleted before BullMQ job removed. Reversed order = free PDF bug.
5. activatePack idempotent. Must be safe to call twice with the same arguments.
6. isPaid, score, pdfR2Key, packId never in user-facing Zod schemas. Server-only fields.
7. npm run test:scoring after any scoring change. Wrong scores fail silently.
8. Never pm2 restart on VPS. Always stop + delete + start.
9. CF-IPCountry for currency detection, never request body. Body country = revenue exploit.
10. Scoring engine is a pure function. No external calls, no randomness inside the engine itself.

Questions and code in this guide reflect ScoreResume's actual implementation. Numbers and configs may be updated as the product evolves.