What this chapter does to you: Most people treat "SEO" as one thing — a bag of tricks to make Google like you. It isn't one thing. It's three separate systems stacked on top of each other — crawling, indexing, and ranking — and each one has its own failure mode, independent of the others. A page can be crawled and never indexed. A page can be indexed and never rank. After this chapter you'll know which of the three you're actually debugging, instead of throwing "SEO fixes" at a problem that isn't where you think it is.
Three systems, not one
Every one of those dotted branches is a different bug with a different fix. "My page isn't ranking" could mean it was never crawled, was crawled but excluded from the index, or is indexed but losing on relevance — and the fix for each is completely different. Diagnosing which stage you're stuck at is step one, always, before touching a single word of content.
Stage 1 — Discovery & crawling
A crawler (Googlebot, Bingbot, and now a growing list of AI crawlers like
GPTBot, ClaudeBot, PerplexityBot) finds URLs three ways: links from pages it
already knows about, your sitemap, and URLs it's crawled before. robots.txt
controls whether a crawler is allowed to fetch a URL at all — it's a
permission gate, not an indexing instruction. This is the single most
common confusion in technical SEO:
| Mechanism | Controls | Common mistake |
|---|---|---|
robots.txt (Disallow) | Whether a crawler fetches the URL | Using it to try to "remove" a page from search — a disallowed page can still get indexed (with no content) if enough external links point to it |
<meta name="robots" content="noindex"> | Whether a fetched page enters the index | Requires the page to be crawlable — if robots.txt blocks it, the crawler never sees the noindex tag to obey it |
canonical tag | Which URL among duplicates is "the real one" | Pointing every variant to itself instead of consolidating duplicate/near-duplicate pages |
If you want a page truly out of search, noindex it and allow crawling
so the instruction is actually seen — this is the opposite of the instinct
most people have.
Stage 2 — Rendering: the part that quietly breaks AI visibility
This is the stage most engineering teams get wrong without realizing it, and it's the one most directly under your control as the person writing the frontend.
Googlebot renders JavaScript — but in a second, deferred pass, sometimes minutes to days after the initial crawl, and only if the first pass decided the page was worth the extra compute. Bing renders JS less reliably. And the crawlers that matter for GEO — GPTBot, ClaudeBot, PerplexityBot, and most other AI crawlers mostly do not execute JavaScript at all. They fetch the raw HTML response and read what's there.
That means: a client-side-rendered SPA can be structurally invisible to
most AI answer engines, even if Google eventually indexes it after a
delayed render pass. If your page's content only exists after useEffect
fires a fetch call, an AI crawler sees an empty <div id="root"> and
nothing else.
This is exactly why the reader app you're using right now renders its markdown content as a server component — the full HTML, headings, and text are already in the initial response, before any JavaScript runs. No crawler, JS-executing or not, ever sees a blank page. This one architectural choice — SSR/SSG over pure CSR — is the highest-leverage technical SEO decision you will make, because it's the one thing that determines whether you're visible to every crawler category at once, not just Google's.
Stage 3 — Indexing: crawled ≠ included
Once a page is fetched and rendered, the search engine decides whether it's worth adding to the index at all. Pages get excluded for being:
- Thin — too little unique content to be worth storing and ranking
- Duplicate or near-duplicate — the same content reachable via multiple
URLs (trailing slash vs no trailing slash,
?utm_source=params,wwwvs bare domain) without a canonical tag telling the engine which one wins - Explicitly excluded —
noindex, or blocked before it could be seen
Faceted navigation and pagination are the classic way apps accidentally generate thousands of near-duplicate URLs (every filter combination on a product listing page, for instance) and quietly poison their own index with noise. This is a data-modeling problem as much as an SEO one — it's the same instinct as app 6's cursor-pagination lesson in the Full-Stack Playbook: know exactly which URL is the canonical representation of a piece of content.
Stage 4 — Ranking: happens per query, at request time
Ranking isn't a property of a page — it's computed fresh for every query against the index, weighing (roughly): relevance (does this match the query's intent), quality signals (E-E-A-T, backlinks, engagement), and technical health (Core Web Vitals, mobile-friendliness) as a tiebreaker among otherwise-similar pages. A perfectly indexed, technically flawless page with content that doesn't match what people actually search for will never rank for that query — no amount of technical polish fixes a content-market mismatch.
The mental model to keep
Crawled, indexed, and ranked are three separate funnels, each with its own drop-off, each debugged differently. Before changing anything, ask which stage you're actually stuck at — check Search Console's coverage report to see if a page is indexed at all before you spend a week trying to improve its ranking. The next chapter builds on this foundation to separate three different goals — ranking, being the direct answer, and being cited by an AI — because "getting found" stopped being one game the moment AI answer engines entered the picture.