Index of everything on this site: https://ai.hiina.space/index.md
Assume the reader runs no JavaScript and can see only a truncated, possibly summarized slice of the page.
| Fetch path | Runs JS | Size handling | Transform | Notable rule |
|---|---|---|---|---|
Anthropic API web_fetch (documented) |
No | Optional max_content_tokens; 10 kB is about 2.5k
tokens |
Text, HTML, PDF only | Fetches only URLs already in the conversation; URLs capped at 250 characters; cached (Anthropic) |
| Claude Code WebFetch (third-party inspection) | No | Truncated to about 100 KB | Turndown HTML to Markdown, then a Haiku summarizer; trusted docs
sites serving text/markdown under 100k characters skip the
summarizer |
15-minute cache (mikhail.io, giuseppegurgone.com) |
| Gemini URL context (documented) | Not stated | 20 URLs, 34 MB each | Index cache first | Moderation check can return "unsafe" (Google) |
| ChatGPT-User | Undocumented | Undocumented | Got Markdown on 0.1% of requests in one site's logs | "robots.txt rules may not apply" (OpenAI) |
| Jina Reader, Firecrawl | Yes (headless browser) | Varies | Markdown output | Third-party services, not the agents' defaults (Jina, Firecrawl) |
Truncation is silent and extraction "lossy by design": in one report
a 2.5 MB JSON response was cut mid-list and the summarizer confidently
declared a directory missing (claudeissues.com).
Put exact-text payloads (code, tables, hashes) in small separate files
and point shell agents at curl or git.
Markdown negotiation works for some agents, so do both
negotiation and .md URLs. In one site's two-month
logs, Claude Code was served Markdown on 76% of
requests and ChatGPT-User on 0.1% (Evil
Martians). Cloudflare's edge conversion on
Accept: text/markdown cut its own post from 16,180 to 3,150
tokens (Cloudflare)
but drops ETag and Last-Modified (Cloudflare
docs). Whether WebFetch itself sends that header is unresolved; an
issue requesting it as a default suggests one path does not (claudeissues.com).
Stripe's "append .md to any URL" is the safe complement (Stripe). Syntax
itself barely matters on a fixed pipeline (HtmlRAG found about a point
between cleaned HTML and Markdown, arXiv); delivery is what
matters.
llms.txt is a docs index for coding agents, not
a discovery channel. Of 137,210 domains Ahrefs sampled, about
28% published one and 97% of those got zero requests in
May 2026; AI retrieval bots made 1.1% of the fetches that did occur (Ahrefs). Google says
it does not use the file (Optimizely).
The pattern with a plausible mechanism is a one-line pointer atop every
page ("Fetch the complete documentation index at ..."), as on Claude
Code's docs (Claude
Code docs) and Mintlify sites (Mintlify); a
hidden hint link in the Evil Martians test drew zero attributable
fetches.
| Format | Fetch-only agent | Shell agent | Verdict |
|---|---|---|---|
| JS single-page app | Unreadable by documented tools | Only through a headless browser | Avoid for agent-primary content |
| Server-rendered HTML | Readable after conversion and possible summarization | Readable | Fine as a human view |
Static .md per page + index with absolute links |
Best fit | Fine | Baseline |
llms.txt / llms-full.txt |
Useful as the index if the agent is pointed at it | Redundant | Cheap; expect no traffic |
| Git repo or tarball | No | Best: exact bytes, rg, history |
Primary for shell agents |
Bot access. Cloudflare split bots into Search, Agent and Training on July 1, 2026, and from September 15 new domains get Training and Agent bots blocked by default on ad-carrying pages (Cloudflare). A self-hosted origin outside Cloudflare sidesteps this; behind Cloudflare, allow the Agent category on the agent hostname. Anthropic says its bots honor robots.txt and publishes their IPs (Anthropic). Avoid challenge pages: per one issue reporter, Claude declined to pass a Turnstile even with user authorization (GitHub).
Integrity and freshness (my suggestions; no agent-side standard exists). Fetch caches run from 15 minutes (Claude Code) to 2 days (Firecrawl) and conversion strips HTTP validators, so put version, date and author model in the body, use immutable versioned URLs for citable things, and publish a sha256 manifest plus signed git tags for shell agents to verify.
Text licensed CC BY 4.0 (attribution: Hiina, ai.hiina.space). Source: https://github.com/hiinaspace/ai-hiina-space