---
id: agent-readers/05-evals-nobody-holds-the-agent-fixed-and
title: 'Agent readers and agent institutions: 5. Evals: nobody holds the agent fixed
  and varies the site'
report: https://ai.hiina.space/reports/agent-readers.md
section: 5
version: 1
updated: '2026-10-26'
author: Hiina
---

Index of everything on this site: https://ai.hiina.space/index.md

# Agent readers and agent institutions: 5. Evals: nobody holds the agent fixed and varies the site

- Section 5 of 11 of the report "Agent readers and agent institutions" by Hiina (v1, updated 2026-10-26)
- Report overview and contents: https://ai.hiina.space/reports/agent-readers.md
- Previous: https://ai.hiina.space/reports/agent-readers/04-agent-site-search-layer-it-by-reader.md
- Next: https://ai.hiina.space/reports/agent-readers/06-inbound-and-bidirectional-make-rejection-cheap-and.md

## 5. Evals: nobody holds the agent fixed and varies the site

**What exists.** BrowseComp-Plus is the template: a frozen 100,195-document corpus, 830 queries, gold documents, mined hard negatives, an oracle condition and search-call counts. Its retriever-only swing of 2x or more on a fixed corpus is exactly the effect size you want to look for, but it varies the retriever, not the site ([arXiv](https://arxiv.org/html/2508.06600v1)). WebWalkerQA has agents traverse real sites (best about 37.5%, falling with depth) ([ACL](https://preview.aclanthology.org/landing_page/2025.acl-long.508.pdf)). Needle in the Web grades difficulty by claim centrality and finds agents at 30% to 33% versus humans at 79% to 86%, with cluttered sites worse ([arXiv](https://arxiv.org/pdf/2512.16553)). The Synthetic Web Benchmark builds miniature internets; one misleading top-ranked article cut GPT-5 from 65.1% to 18.2% ([arXiv](https://arxiv.org/html/2603.00801v1)). Live-web numbers drift: an audit measured Operator's self-reported 87% on WebVoyager at 68.6% ([arXiv](https://arxiv.org/html/2603.29020v1)), and BrowseComp is near saturation on self-reports ([BenchLM](https://www.benchlm.ai/benchmarks/browsecomp)).

**The real gap.** Practitioner tools check delivery, not task success. Agent Reading Test plants canary tokens to catch truncation, JS-only content and HTML/Markdown divergence, and caught agents misreporting their own scores ([Carey](https://dacharycarey.com/2026/04/06/designing-agent-reading-test/)). The 22-check afdocs score's author says its link to task completion is unmeasured; one site scored 100 there and 33 on another checker ([Carey](https://dacharycarey.com/2026/04/18/what-agent-score-can-tell-you/)). GEO, the closest "SEO for agents", measures which source a generator credits (up to 41% gains from citing sources and adding statistics, in a GPT-3.5-era pipeline), not whether an agent finds a fact ([arXiv](https://ar5iv.labs.arxiv.org/html/2311.09735)). No study I found renders one corpus several ways and crosses it with several agents; with about five search wordings tried, call it probably unoccupied.

**A cheap design that separates model navigation from site navigability.** Generate about 1,000 pages of invented entities in the style of agent-written working notes, so answers cannot come from memory. Plant 60 facts at link depths 1 to 4 (half single-hop, half two-page), grade difficulty by claim centrality, add a conflicting near-duplicate per fact and one adversarial top-ranked page. Render the tree as a JS app, server HTML, `.md` plus index, `.md` plus index plus `/search`, git repo, `corpus.db` and MCP. Run a fetch-only API loop, Claude Code, Codex CLI, Gemini CLI and one open-weight model, three runs per valid cell, with closed-book and oracle controls. Grade by exact match and take efficiency from server logs, not self-reports: hops to first gold page, bytes, tool calls, tokens.

Fit accuracy as agent + format + agent×format. The format effect is **site navigability**, the agent effect is **model navigation skill**, and the interaction says which affordances each agent class needs. The cleanest per-site number is **navigability loss**: oracle accuracy minus accuracy under that format for a fixed agent. Released as a harness, this would be the first navigability score validated against task completion.

---

Text licensed CC BY 4.0 (attribution: Hiina, ai.hiina.space). Source: https://github.com/hiinaspace/ai-hiina-space