I filled my agent's wiki with contradictions. It never gave a wrong answer.

There's a comfortable assumption behind a lot of "agent + knowledge base" work: garbage in, garbage out. Feed an agent a messy, stale, duplicate-ridden wiki and it'll confidently tell users the wrong thing. So we invest in dedup, freshness, clean ingestion — to stop the agent from hallucinating.

I built a small testbed to measure that assumption, and it's wrong. Or rather: it's wrong about how bad ingest hurts, and the real answer is more interesting — and harder to catch.

The setup

I built a tiny agent that navigates a wiki the way a person does: it has a
wiki_search tool and a wiki_read tool, it decides what to look up, reads a page, and answers. No vector database, no RAG injection — the agent navigates. (This is increasingly how capable models prefer to work: they know what they need better than a query-time embedding match.)

The wiki is ten markdown pages of deliberately synthetic facts — invented services, made-up numbers the model cannot possibly know from training ("the Orion canary ring holds for 45 minutes before auto-promoting"). That last part matters: because the facts are synthetic, an agent with no wiki genuinely can't answer, so any success is attributable to the wiki, not to the model reciting what it already knew.

First, does the wiki help at all? Baseline (no wiki tools) scored 0/4.
Augmented (wiki tools) scored 4/4. A clean +100-point lift. Good — the testbed works, and the tasks are honest. (the runner)

Then I started degrading the ingest quality and measuring what happened.

The degradation

I built three versions of the wiki:

• clean — the authoritative pages only.
• stale-present — each answer's page now has a contradicting duplicate (a page claiming the canary holds for 30 minutes, not 45), ranked below the real page. This simulates auto-ingest pulling in an old copy.
• stale-outranks — the same contradiction, but keyword-stuffed so it outranks the authoritative page in search. This simulates the very common failure where a spammy or verbose stale doc wins retrieval.

I ran each condition several times per question (agents are stochastic; a single run tells you almost nothing) and measured the rate of confident, correct answers. The result:

clean-answer rate
  clean            100%
  stale-present      8%
  stale-outranks      0%

The wiki's usefulness collapses as ingest quality drops. Ingest quality clearly matters enormously. But here's the part that changed how I think about it.

The agent never gave a wrong answer

Not once. I went and read the traces expecting to find the agent confidently reciting "30 minutes." Instead, in the degraded runs it did this:

The wiki has conflicting information: [deploy-orion] says the canary ring holds for 45 minutes, while [orion-canary-ring-guide] says 30 minutes.

It read both pages, noticed they disagreed, and refused to pick. It hedged. A capable model is remarkably good at not being confidently wrong — it detects the contradiction and hands it back to you.

So "bad ingest makes the agent hallucinate" is the wrong model. The real cost is subtler, and worse in a way that's easy to miss:

1. The wiki stops being authoritative. The entire point of a wiki is that you ask a question and get one trusted answer. With contradictory ingest, you ask "how long is the canary hold?" and get "the wiki says 45 or 30 — unclear." The value proposition is destroyed, even though nothing is technically "wrong."
2. Every query costs more. The agent that would have read one page now reads several, reasons about which to trust, and writes a longer answer. Multiply that across a busy system.
3. The failure is invisible without ground truth. A hedged answer looks reasonable. Nobody files a bug for "the agent was slightly too careful." You only see the collapse if you have known-correct answers to grade against — which is exactly what this testbed has and a production system usually doesn't.

And a caveat that makes it worse, not better: I tested a strong model. A weaker one — or a subtler contradiction than "45 vs 30" — is exactly where the hedge turns into a confident wrong answer. The robustness I measured is a property of this model, not a guarantee.

Ranking is the lever

Notice the gap between the two degraded conditions: 8% when the stale page ranked below the real one, 0% when it outranked it. That's the actionable part. A capable navigation agent reads the top results and tends to stop — so what decides your outcome is whether the authoritative page wins retrieval. Bad ingest hurts most when it lets a stale or duplicate page rank above the real one.

Which reframes the ingest-quality job. It isn't "clean everything." It's the specific things that keep the authoritative page on top: dedup (so there's no stale copy to compete), staleness detection and removal, and recency or authority signals in ranking. Prettifying pages the agent never reads is wasted effort; keeping the wrong page out of the #1 slot is the whole game.

The part that almost fooled me

My first grader checked whether the answer contained the right value — did "45" appear anywhere? By that measure, every condition scored 100%. No degradation at all. I nearly concluded the agent was invincible.

It scored 100% because a hedge contains the right value too ("the wiki says 45 or 30"). The lenient grader counted "45 or 30, unclear" as a win. Only when I changed the metric to a confident clean answer — the right value present and the stale value absent — did the collapse from 100% to 0% appear.

This is the quiet lesson under the loud one: the metric you pick decides whether you can see the problem at all. "Does the right token appear?" is the easy thing to measure and it would have told me, confidently, that ingest quality doesn't matter. For agent knowledge systems, measure the thing users actually care about — a clean, trustworthy answer — not the presence of a substring.

The takeaway

If you're building a knowledge base or memory system for agents, don't reach for the "it'll hallucinate" argument to justify ingest quality — capable agents largely won't, and a skeptic will call your bluff. Reach for the real one:

Bad ingest doesn't make your agent lie. It turns clean, authoritative answers into hedges, makes every query cost more, and hides a wrong-answer risk that surfaces the moment your model is a little weaker or the conflict a little subtler. And you won't see any of it unless you measure clean-answer rate against ground truth.

Bad ingest doesn't cause wrong answers. It destroys clean ones — quietly.

*Built as a learning project exploring agent memory / knowledge systems. The navigation-wiki testbed, the three wiki conditions, and the full results are in zachzwy/agentloop —eval/wiki-eval.js runs it, and eval/wiki-findings.md has the numbers.

I'm Wenyu — github | linkedin.