An AI answer is not generated from the whole web. It is generated from what the system managed to fetch, split, score and hand to the model. Understanding that pipeline explains most of the “why did it cite them and not me” questions.

1. Fetch

Before ranking, the engine has to have your page. That means a crawler with a user agent identifiable in your logs, an HTTP response that is not a challenge page, and HTML that contains the content.

What breaks here:

  • A 403 from a WAF or bot rule aimed at “unknown” user agents.
  • Content rendered only after client-side fetches, so the HTML arrives empty.
  • Rate limits tuned for humans, so a burst of page fetches times out.
  • robots.txt that disallows the path you want quoted.

Check the access log rather than assuming. A page that returns 200 to you and 403 to a crawler looks perfectly fine from a browser.

2. Split

Pages are chunked into passages, and each passage is embedded and retrieved separately. Chunk boundaries rarely follow your intentions: a heading plus three paragraphs may become one chunk; a long code block may become another; a table may survive or may not.

Two consequences:

  • Self-contained beats clever. A passage that opens with “As mentioned above” loses its meaning once separated from the section above it.
  • Headings are chunk titles. The text of a heading is often how a passage is described internally. “Pricing” tells the retriever nothing; “How much does the API cost in 2026” tells it a lot.

3. Score

Retrieval scores a passage against a user question that may not share a single keyword with your page. Three things move that score:

  1. Lexical overlap with the question — useful, but weaker than people assume.
  2. Semantic closeness of the passage as a whole.
  3. Source-level priors: is this site known for this topic, is the page dated, does it appear in other citations of the same conversation.

That third bucket is why a small site can beat a big one on a narrow question, and why a big site rarely wins a question it has never covered.

4. Ground

Given several candidate passages, the model is asked to write an answer supported by them. A passage that states its claim plainly, with its conditions attached, is easier to use than one that hedges for eight lines. So:

  • Lead with the claim, then the caveat — not the other way round.
  • Put the numbers where the sentence is, not in a PDF linked at the bottom.
  • Name the thing you are talking about. “The theme” means nothing outside your page; “the Inkfolio Astro theme” survives extraction.

5. Attribute

Citations go to sources the system can identify and link. Practically, pages that get cited tend to share these habits:

  • One idea per heading, with the answer in the first sentence.
  • Real publication and update dates, matching the content.
  • An identifiable author or organisation, with a page that exists.
  • Canonical URLs and no near-duplicate variants competing with each other.

A table you can test against

Failure What the engine sees Fix
Content behind JS Empty HTML Server-render, or ship the text in the initial response
Bot blocked 403 in your log Allow the crawler, verify with a fetch using its user agent
Vague headings Chunk labelled “Overview” Headings that state the question
Cross-references Passage without context Restate the subject in each section
Duplicate pages Several versions of the same text Canonicals, one URL per idea

Honest limits

Engines do not publish their retrieval weights, and they change them. Anything presented as a fixed formula — a percentage lift, a “GEO score” — is a guess dressed as a metric. What is stable is the shape: fetchable HTML, self-contained passages, specific claims, clear attribution.