---
id: agent-readers/04-agent-site-search-layer-it-by-reader
title: 'Agent readers and agent institutions: 4. Agent site search: layer it by reader
  capability'
report: https://ai.hiina.space/reports/agent-readers.md
section: 4
version: 1
updated: '2026-10-26'
author: Hiina
---

Index of everything on this site: https://ai.hiina.space/index.md

# Agent readers and agent institutions: 4. Agent site search: layer it by reader capability

- Section 4 of 11 of the report "Agent readers and agent institutions" by Hiina (v1, updated 2026-10-26)
- Report overview and contents: https://ai.hiina.space/reports/agent-readers.md
- Previous: https://ai.hiina.space/reports/agent-readers/03-publishing-for-agent-readers-static-markdown-small.md
- Next: https://ai.hiina.space/reports/agent-readers/05-evals-nobody-holds-the-agent-fixed-and.md

## 4. Agent site search: layer it by reader capability

**There is no standard "site search for agents", and browser search engines are out.** Pagefind is queryable only through a browser JS/WebAssembly API; its Node and Python APIs only build indexes ([Pagefind](https://pagefind.app/docs/api/), [Node API](https://pagefind.app/docs/node-api/)). Stork is winding down ([Stork](https://github.com/jameslittle230/stork)) and docfind documents browser use only ([docfind](https://github.com/microsoft/docfind)).

**For shell agents, a SQLite FTS5 file is the server-free answer.** FTS5 ships `bm25()`, `snippet()` and a trigram tokenizer, though a distro `sqlite3` may lack FTS5, so check the agent images ([SQLite](https://sqlite.org/fts5.html)). HTTP Range tricks like sql.js-httpvfs matter only for databases too big to download (a full-text query on an 8 MB FTS table fetched about 70 KiB, [phiresky](https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/)); a Markdown corpus rarely is.

**For fetch-only agents, search must fit the URL-in-context rule.** Anthropic's API fetch tool only fetches URLs already in the conversation, so a model-built `?q=foo` may be refused; the docs do not address query strings ([Anthropic](https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-fetch-tool)). The safe design: precomputed result pages linked from every page (`/topics/x.md`, `/recent.md`) plus a `/search?q=` endpoint returning absolute links, so every follow-up fetch is in context. Datasette already offers `?_search=` and read-only `?sql=` as JSON over GET with 1,000-row and 1-second defaults ([Datasette](https://docs.datasette.io/en/stable/json_api.html)). I found no site serving search results as `text/markdown` for agents.

**Docs vendors are converging on remote MCP.** Mintlify gives every site `/mcp` with search, a shell-style "virtual filesystem" tool and a feedback tool ([Mintlify](https://www.mintlify.com/docs/ai/model-context-protocol)); GitBook ([GitBook](https://gitbook.com/docs/~gitbook/embed/page/ai-and-search/mcp-servers-for-published-docs)), GitMCP ([GitMCP](https://github.com/idosal/git-mcp)) and Cloudflare AI Search (site must be a zone on the publisher's account, [Cloudflare](https://blog.cloudflare.com/ai-search-easier)) do similar. The only quality data is Context7's internal benchmark of its own redesign: 65% fewer context tokens, "slight" quality gains ([Upstash](https://upstash.com/blog/new-context7)). MCP only reaches harnesses whose operator configured it, so layer it on URL-addressable search.

**Embeddings belong behind the publisher's endpoint.** sqlite-vec ([sqlite-vec](https://alexgarcia.xyz/sqlite-vec/)) and Willison's `llm`, which records the model per collection ([Simon Willison](https://simonwillison.net/2023/Sep/4/llm-embeddings/)), make shipping vectors possible, but readers must embed queries with the same model, which most sandboxes cannot. Hybrid retrieval inside `/search` gets the BrowseComp-Plus recall gain with no reader-side coupling.

| Interface | Fetch-only agent | Shell agent | Server needed | Evidence |
| --- | --- | --- | --- | --- |
| `.md` pages + index + topic pages + sitemap | Yes (many hops) | Yes | No | Documented fetch behavior |
| Tarball / `git clone --depth 1` | No | Yes (`rg`) | No | Grep strong on structured corpora |
| `corpus.db` FTS5 download | No | Yes, if FTS5 is compiled in | No | Documented SQLite features; agent use unmeasured |
| `/search?q=` returning Markdown, hybrid behind it | Yes, if the URL pattern is in context | Yes (`curl`) | Small service | No precedent found |
| Datasette `?_search=` JSON | Yes, same caveat | Yes | Python process | Documented API |
| MCP `search` + `read_page` | Only where the harness has it configured | Same | Small service | Vendors ship it; no usage data |
| Shipped embeddings file | No | Only with the same model available | No | Tooling exists; no established practice |
| Pagefind / tinysearch / docfind | No | No documented path | No | Browser-only |

---

Text licensed CC BY 4.0 (attribution: Hiina, ai.hiina.space). Source: https://github.com/hiinaspace/ai-hiina-space