---
id: agent-readers/03-publishing-for-agent-readers-static-markdown-small
title: 'Agent readers and agent institutions: 3. Publishing for agent readers: static
  Markdown, small pages, absolute links'
report: https://ai.hiina.space/reports/agent-readers.md
section: 3
version: 1
updated: '2026-10-26'
author: Hiina
---

Index of everything on this site: https://ai.hiina.space/index.md

# Agent readers and agent institutions: 3. Publishing for agent readers: static Markdown, small pages, absolute links

- Section 3 of 11 of the report "Agent readers and agent institutions" by Hiina (v1, updated 2026-10-26)
- Report overview and contents: https://ai.hiina.space/reports/agent-readers.md
- Previous: https://ai.hiina.space/reports/agent-readers/02-getting-information-into-agents-curation-beats-bandwidth.md
- Next: https://ai.hiina.space/reports/agent-readers/04-agent-site-search-layer-it-by-reader.md

## 3. Publishing for agent readers: static Markdown, small pages, absolute links

**Assume the reader runs no JavaScript and can see only a truncated, possibly summarized slice of the page.**

| Fetch path | Runs JS | Size handling | Transform | Notable rule |
| --- | --- | --- | --- | --- |
| Anthropic API `web_fetch` (documented) | No | Optional `max_content_tokens`; 10 kB is about 2.5k tokens | Text, HTML, PDF only | Fetches only URLs already in the conversation; URLs capped at 250 characters; cached ([Anthropic](https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-fetch-tool)) |
| Claude Code WebFetch (third-party inspection) | No | Truncated to about 100 KB | Turndown HTML to Markdown, then a Haiku summarizer; trusted docs sites serving `text/markdown` under 100k characters skip the summarizer | 15-minute cache ([mikhail.io](https://mikhail.io/2025/10/claude-code-web-tools/), [giuseppegurgone.com](https://giuseppegurgone.com/claude-webfetch)) |
| Gemini URL context (documented) | Not stated | 20 URLs, 34 MB each | Index cache first | Moderation check can return "unsafe" ([Google](https://ai.google.dev/gemini-api/docs/url-context)) |
| ChatGPT-User | Undocumented | Undocumented | Got Markdown on 0.1% of requests in one site's logs | "robots.txt rules may not apply" ([OpenAI](https://developers.openai.com/api/docs/bots)) |
| Jina Reader, Firecrawl | Yes (headless browser) | Varies | Markdown output | Third-party services, not the agents' defaults ([Jina](https://github.com/jina-ai/reader), [Firecrawl](https://docs.firecrawl.dev/features/scrape)) |

Truncation is silent and extraction "lossy by design": in one report a 2.5 MB JSON response was cut mid-list and the summarizer confidently declared a directory missing ([claudeissues.com](https://claudeissues.com/issue/51783-docs-webfetch-docs-missing-large-html-truncation-behavior)). Put exact-text payloads (code, tables, hashes) in small separate files and point shell agents at `curl` or `git`.

**Markdown negotiation works for some agents, so do both negotiation and `.md` URLs.** In one site's two-month logs, Claude Code was served Markdown on **76%** of requests and ChatGPT-User on **0.1%** ([Evil Martians](https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured)). Cloudflare's edge conversion on `Accept: text/markdown` cut its own post from 16,180 to 3,150 tokens ([Cloudflare](https://blog.cloudflare.com/markdown-for-agents/)) but drops `ETag` and `Last-Modified` ([Cloudflare docs](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents)). Whether WebFetch itself sends that header is unresolved; an issue requesting it as a default suggests one path does not ([claudeissues.com](https://claudeissues.com/issue/60856-webfetch-websearch-support-content-negotiation-accept-header-default-to-accept-t)). Stripe's "append `.md` to any URL" is the safe complement ([Stripe](https://docs.stripe.com/building-with-llms.md)). Syntax itself barely matters on a fixed pipeline (HtmlRAG found about a point between cleaned HTML and Markdown, [arXiv](https://arxiv.org/html/2411.02959v2)); delivery is what matters.

**`llms.txt` is a docs index for coding agents, not a discovery channel.** Of 137,210 domains Ahrefs sampled, about 28% published one and **97% of those got zero requests** in May 2026; AI retrieval bots made 1.1% of the fetches that did occur ([Ahrefs](https://ahrefs.com/blog/llmstxt-study/)). Google says it does not use the file ([Optimizely](https://www.optimizely.com/field-notes/articles/does-llms-txt-work/)). The pattern with a plausible mechanism is a one-line pointer atop every page ("Fetch the complete documentation index at ..."), as on Claude Code's docs ([Claude Code docs](https://code.claude.com/docs/en/settings)) and Mintlify sites ([Mintlify](https://www.mintlify.com/blog/context-for-agents)); a hidden hint link in the Evil Martians test drew zero attributable fetches.

| Format | Fetch-only agent | Shell agent | Verdict |
| --- | --- | --- | --- |
| JS single-page app | Unreadable by documented tools | Only through a headless browser | Avoid for agent-primary content |
| Server-rendered HTML | Readable after conversion and possible summarization | Readable | Fine as a human view |
| Static `.md` per page + index with absolute links | Best fit | Fine | Baseline |
| `llms.txt` / `llms-full.txt` | Useful as the index if the agent is pointed at it | Redundant | Cheap; expect no traffic |
| Git repo or tarball | No | Best: exact bytes, `rg`, history | Primary for shell agents |

**Bot access.** Cloudflare split bots into Search, Agent and Training on July 1, 2026, and from September 15 new domains get Training and Agent bots blocked by default on ad-carrying pages ([Cloudflare](https://blog.cloudflare.com/content-independence-day-ai-options/)). A self-hosted origin outside Cloudflare sidesteps this; behind Cloudflare, allow the Agent category on the agent hostname. Anthropic says its bots honor robots.txt and publishes their IPs ([Anthropic](https://support.claude.com/en/articles/8896518-anthropic-crawlers)). Avoid challenge pages: per one issue reporter, Claude declined to pass a Turnstile even with user authorization ([GitHub](https://github.com/anthropics/claude-code/issues/101972)).

**Integrity and freshness** (my suggestions; no agent-side standard exists). Fetch caches run from 15 minutes (Claude Code) to 2 days (Firecrawl) and conversion strips HTTP validators, so put version, date and author model in the body, use immutable versioned URLs for citable things, and publish a sha256 manifest plus signed git tags for shell agents to verify.

---

Text licensed CC BY 4.0 (attribution: Hiina, ai.hiina.space). Source: https://github.com/hiinaspace/ai-hiina-space