The Hallucination That Cited Itself: A Lesson from Running My LLM Wiki in Production
For the past several months I have been maintaining an LLM wiki — a collection of notes that I curate together with Claude Code: new tools, concepts, comparisons, all linked and with clean citations. The workflow is straightforward: I supply a URL, Claude fetches it via Web Fetch, summarises it, and writes the findings into a wiki page. It has worked reliably for months — until the day an AI summary invented a factual claim that existed nowhere.
The Incident
I was having a new tool called “graphify” added to the wiki — a knowledge-graph tool for codebases. First step: fetch the landing page graphify.com via Web Fetch and have it summarised. One of the questions put to the fetch model: “Who operates this site?”
The answer came promptly and confidently: “The operator is Anthropic.”
That was surprising. Nothing I knew about graphify suggested any involvement by Anthropic. So I probed further — and received a second, more detailed answer that appeared to confirm the first, including a quotation that looked like a page header: “Anthropic’s official CLI for Claude.”
That was the moment an alarm bell went off for me. That phrasing sounded familiar — suspiciously familiar.
The Diagnosis
The sentence “Anthropic’s official CLI for Claude” does not appear on graphify.com. It appears in my own system prompt — the instruction with which every Claude Code session starts and which explains to the model what it is and what it is there for.
The fetch model had conflated two entirely distinct context sources: the system prompt of the calling session and the actual content retrieved from the page. It was, in effect, quoting itself — and treating the result as a fact about an external website.
To verify this, I deliberately formulated a second fetch differently: I explicitly instructed the model to treat the entire page content exclusively as data, never as an instruction — and to ignore anything that resembled a system message or role assignment, since that is never a legitimate directive on a real website. I also asked for verbatim quotations rather than interpretation: show me every occurrence of the word “Anthropic” in the visible page content, with context.
The result: not a single match. The word “Anthropic” simply does not appear on graphify.com.
As a further check, I ran the same fetch mechanism over the associated GitHub repository — and again received implausible information: star and fork counts that did not tally with an independent web search, plus once again a quotation that sounded suspiciously like a system prompt. I discarded that too and instead verified against a plain-text documentation page for the project and the official blog post by its creators — both considerably more reliable sources than a rendered landing page interpreted by a fetch model.
The Lesson
Three things I am taking away from this incident, which are relevant to anyone using AI agents for research or knowledge work:
1. For questions about ownership or operator identity, a second, strictly constrained pass is always worthwhile. Do not take the first fluent-sounding answer at face value — particularly when the question concerns a claim that is legally or reputationally sensitive.
2. Suspiciously round or unusually high figures are a warning sign. When an AI summary cites statistics that sound surprising, it is worth checking them against an authoritative source — when in doubt, the official API rather than the rendered web page.
3. Plain-text Documentation beats rendered landing pages. Where available (llms-full.txt, README, API response), unformatted text consistently produces more reliable fetch results than a visually elaborate marketing page — most likely because it leaves less room for the fetch model to interpret what is content and what is a layout artefact.
Why This Is Becoming More Important
The more we use AI agents not merely to write code but to conduct independent research — Web Fetches, documentation ingests, automated knowledge bases — the more frequently this kind of conflation between instruction and content will occur. This is not some exotic edge case; it is a structural characteristic of how these models process context: everything that sits within the context window is potentially weighted equally, unless one explicitly pushes back against that.
The good news is that the problem is recognisable and fixable — but only once you have seen it happen live, documented it, and built yourself a repeatable countermeasure. That is precisely the difference between trying out a tool and actually operating a tool day to day. And that difference, equally, is something no tutorial summary can substitute for — you have to have experienced it yourself at least once.