An AI answer is not generated from the whole web. It is generated from what the system managed to fetch, split, score and hand to the model. Understanding that pipeline explains most of the “why did it cite them and not me” questions.
1. Fetch
Before ranking, the engine has to have your page. That means a crawler with a user agent identifiable in your logs, an HTTP response that is not a challenge page, and HTML that contains the content.
What breaks here:
- A
403from a WAF or bot rule aimed at “unknown” user agents. - Content rendered only after client-side fetches, so the HTML arrives empty.
- Rate limits tuned for humans, so a burst of page fetches times out.
robots.txtthat disallows the path you want quoted.
Check the access log rather than assuming. A page that returns 200 to you and 403 to a
crawler looks perfectly fine from a browser.
2. Split
Pages are chunked into passages, and each passage is embedded and retrieved separately. Chunk boundaries rarely follow your intentions: a heading plus three paragraphs may become one chunk; a long code block may become another; a table may survive or may not.
Two consequences:
- Self-contained beats clever. A passage that opens with “As mentioned above” loses its meaning once separated from the section above it.
- Headings are chunk titles. The text of a heading is often how a passage is described internally. “Pricing” tells the retriever nothing; “How much does the API cost in 2026” tells it a lot.
3. Score
Retrieval scores a passage against a user question that may not share a single keyword with your page. Three things move that score:
- Lexical overlap with the question — useful, but weaker than people assume.
- Semantic closeness of the passage as a whole.
- Source-level priors: is this site known for this topic, is the page dated, does it appear in other citations of the same conversation.
That third bucket is why a small site can beat a big one on a narrow question, and why a big site rarely wins a question it has never covered.
4. Ground
Given several candidate passages, the model is asked to write an answer supported by them. A passage that states its claim plainly, with its conditions attached, is easier to use than one that hedges for eight lines. So:
- Lead with the claim, then the caveat — not the other way round.
- Put the numbers where the sentence is, not in a PDF linked at the bottom.
- Name the thing you are talking about. “The theme” means nothing outside your page; “the Inkfolio Astro theme” survives extraction.
5. Attribute
Citations go to sources the system can identify and link. Practically, pages that get cited tend to share these habits:
- One idea per heading, with the answer in the first sentence.
- Real publication and update dates, matching the content.
- An identifiable author or organisation, with a page that exists.
- Canonical URLs and no near-duplicate variants competing with each other.
A table you can test against
| Failure | What the engine sees | Fix |
|---|---|---|
| Content behind JS | Empty HTML | Server-render, or ship the text in the initial response |
| Bot blocked | 403 in your log | Allow the crawler, verify with a fetch using its user agent |
| Vague headings | Chunk labelled “Overview” | Headings that state the question |
| Cross-references | Passage without context | Restate the subject in each section |
| Duplicate pages | Several versions of the same text | Canonicals, one URL per idea |
Honest limits
Engines do not publish their retrieval weights, and they change them. Anything presented as a fixed formula — a percentage lift, a “GEO score” — is a guess dressed as a metric. What is stable is the shape: fetchable HTML, self-contained passages, specific claims, clear attribution.