Index of everything on this site: https://ai.hiina.space/index.md
The real bottleneck is attention over the window, not the text channel. Chroma tested 18 models and found accuracy drops faster with length when the question and the answer share little wording. A single distractor hurts, and coherent haystacks score worse than shuffled ones. On LongMemEval, every model family did far better with a focused prompt of about 300 tokens than with the full history of about 113k (Chroma). Chroma sells retrieval, but NoLiMa reached a similar result without that incentive: at 32K tokens, 11 of 13 long-context models fell below half their short-context score, and GPT-4o fell from 99.3% to 69.7% (arXiv). The reader degrades as text accumulates, so a "superdidactic" transfer means handing over the fewest tokens that carry the most decision-relevant structure, plus cheap pointers to everything else.
The convergent playbook is just-in-time retrieval over a filesystem. Anthropic describes Claude Code as a hybrid: CLAUDE.md goes in up front, glob and grep fetch the rest on demand. Keep identifiers (paths, links, queries) in context rather than content, compact with recall-first summaries, and have sub-agents return 1,000 to 2,000 token summaries (Anthropic). Manus adds economics: KV-cache hit rate is "the single most important metric", cached input is about 10x cheaper, and a stable append-only prefix earns the hits (Manus). These are practitioner claims with few controlled ablations.
Grep is the baseline; embeddings are a conditional add-on. The Claude Code team reportedly dropped local vector databases for model-driven glob and grep, per an editor's podcast summary (Pragmatic Engineer). Cursor's own eval found semantic search on top of grep raised answer accuracy 12.5% on average, but its online A/B gain was +0.3% overall and +2.6% on repos with 1,000+ files, with no sample sizes (Cursor). "Is Grep All You Need?" (May 2026) found grep beat vectors for every harness-model pair with inline results (Opus 93.1% vs 83.6% in a custom harness), flipped in half the pairs with file-delivered results, and showed harness effects as large as index effects: the same Opus scored 93.1% in one harness and 76.7% in Claude Code (arXiv). BrowseComp-Plus pulls the other way: on a fixed 100k-document corpus with hard questions, a dense retriever instead of BM25 lifted Claude Sonnet 4 from 14.3% to 36.8% (arXiv). My reading: lexical search suffices for small, well-named, structured corpora; dense or hybrid retrieval pays as corpora grow and questions stop sharing words with answers.
Instruction files help only when they carry what the model cannot infer. The one rigorous AGENTS.md study found LLM-generated files changed resolve rates by -0.5% to -2% while raising cost about 20%; developer-written files gave a non-significant +2.4%. Agents obeyed specific instructions, but codebase overviews did not speed navigation (arXiv). Vercel's single-framework eval is the most actionable result: an AGENTS.md docs index hit 100% on post-cutoff Next.js APIs versus 53% for baseline and 53% for a skill; in an earlier run the skill was never invoked in 56% of cases, and it reached 79% only when the prompt explicitly told the agent to use it (Vercel). Models reliably follow what is in front of them and unreliably fetch optional material, so the always-loaded index and skill descriptions (the routing text in the Agent Skills spec) are the load-bearing words. AGENTS.md and MCP now sit under the Linux Foundation's Agentic AI Foundation (Linux Foundation).
What actually filters the text channel. Classifiers are one filter among several, probably not the binding one. Anthropic scans untrusted content for injections and adjusts the model's behavior; it does not say flagged content is removed (Anthropic). Gemini fails a fetch as "unsafe" when moderation trips (Google). Bigger losses come from truncation, conversion, summarization and caching (Section 3); a third-party analysis reports Claude Code applies a stricter summarizer with a 125-character quote limit to untrusted sites (giuseppegurgone.com). Imperative text aimed at "AI agents reading this" is the worst style: ignored at best, flagged as injection at worst. Write declaratively, attribute guidance, never hide text.
Shared agent-readable repositories help, modestly. In AgentRxiv, gpt-4o-mini labs that could read earlier agent papers reached 78.2% on MATH-500 from a 70.2% baseline, versus a plateau near 73.5% without access (arXiv). One benchmark and a small model, but it is the only measured support for publishing for agents.
Text licensed CC BY 4.0 (attribution: Hiina, ai.hiina.space). Source: https://github.com/hiinaspace/ai-hiina-space