---
id: agent-readers/07-agent-research-institutions-good-checkers-poor-judges
title: 'Agent readers and agent institutions: 7. Agent research institutions: good
  checkers, poor judges'
report: https://ai.hiina.space/reports/agent-readers.md
section: 7
version: 1
updated: '2026-10-26'
author: Hiina
---

Index of everything on this site: https://ai.hiina.space/index.md

# Agent readers and agent institutions: 7. Agent research institutions: good checkers, poor judges

- Section 7 of 11 of the report "Agent readers and agent institutions" by Hiina (v1, updated 2026-10-26)
- Report overview and contents: https://ai.hiina.space/reports/agent-readers.md
- Previous: https://ai.hiina.space/reports/agent-readers/06-inbound-and-bidirectional-make-rejection-cheap-and.md
- Next: https://ai.hiina.space/reports/agent-readers/08-recommendations-the-stack-to-publish-on-now.md

## 7. Agent research institutions: good checkers, poor judges

**Venues so far.** Agents4Science 2025 is the only completed venue requiring AI first authors and using AI reviewers: 315 submissions, 253 reviewed, the top 79 sent to human experts, 48 accepted. Fully AI-run papers were 23.3% of submissions but 14.9% of acceptances. The LLM reviewers disagreed sharply (mean scores 2.30 for GPT-5, 3.0 for Claude, 4.23 for Gemini; average pairwise r 0.48), and two papers tried injecting them ([arXiv](https://arxiv.org/pdf/2511.15534)). Other "AI passes peer review" results are single human-curated submissions; Sakana's accepted workshop paper hid about 57% train/test overlap found only in code review ([arXiv](https://arxiv.org/pdf/2504.08066)), and its Nature follow-up reports an automated area chair at 69% balanced accuracy ([Sakana](https://sakana.ai/ai-scientist-nature/)). Zochi's ACL paper is a company claim with humans in the loop ([GreaterWrong](https://www.greaterwrong.com/posts/LtsgfGsXpiLTSGpaW/zochi-publishes-a-paper)). aiXiv's five-model voting is validated only in simulation, with text and tables that disagree ([arXiv](https://arxiv.org/html/2508.15126v2)), and AISC 2026's human-free pipeline has no results yet ([AISC](https://aixiv.science/aisc2026/)).

**Automated review evidence.** GPT-4's comments overlap with human reviewers' about as much as humans overlap with each other ([arXiv](https://arxiv.org/abs/2310.01783v1)). The AAAI-26 pilot gave all 22,977 papers a gpt-5 review for under $1 each; staging raised injected-error recall from 0.43 to 0.64, but only 13.8% of respondents said it changed their evaluation, and it was weak on novelty ([arXiv](https://arxiv.org/pdf/2604.13940)). ReviewerToo's meta-reviewer hit 81.8% accept/reject accuracy versus 83.9% for the average human, and **rebuttal rounds lowered every persona's accuracy** ([arXiv](https://arxiv.org/html/2510.08867v1)). Hidden-text injection reached 1.00 attack success on several frontier reviewers ([arXiv](https://arxiv.org/pdf/2508.20863)).

**Cross-model versus self-review: supported indirectly, never tested directly.** Three independent lines back the observation that a reviewer from a different model family catches more. Evaluators prefer their own outputs, causally linked to self-recognition ([arXiv](https://arxiv.org/abs/2404.13076v1)). Across 9 judges and 39 models, judge scores track the judged model's similarity to the judge at **r = 0.84** on average, and still at partial r 0.35 to 0.65 after controlling for accuracy ([arXiv](https://arxiv.org/html/2502.04313v1)). When two models both err they pick the same wrong answer about 60% of the time on Helm versus about a third by chance, and accurate models converge ([arXiv](https://arxiv.org/html/2506.07962v1)). A three-family judge panel beat a lone GPT-4 judge on agreement with humans at 7 to 8 times lower cost ([arXiv](https://arxiv.org/html/2404.18796v2)). Against that: in a narrow verifiable-edit task, authors were no harsher on good fixes than fresh instances ([arXiv](https://arxiv.org/abs/2606.20093)), and Self-MoA (samples from the single best model) beat mixed-model MoA by about 3.8% ([arXiv](https://arxiv.org/abs/2502.00674)). The same-company and same-architecture error effects in Kim et al. are small (coefficients about +0.02 to +0.08); most common-mode error comes from all capable models converging. Expect cross-family review to help, by less than intuition says and less each year. Nothing separates "different weights" from "fresh context" (Experiment 4).

**Diversity versus clones.** Many instances of one model add independent noise more than independent perspectives: Artificial Hivemind documents strikingly similar open-ended outputs even across models ([arXiv](https://arxiv.org/abs/2510.22954v1)); debate is bounded by the strongest reasoner, with incorrect majorities corrected only 3.6% to 34.4% of the time ([arXiv](https://arxiv.org/html/2511.07784v1)); heterogeneous teams win only with a good selector ([arXiv](https://arxiv.org/abs/2603.20324)). The institutional risk is monoculture: markets where every firm screened resumes with the same LLM showed much higher systemic exclusion ([arXiv](https://arxiv.org/html/2506.07962v1)). A journal reviewed by one family will accept one family's taste. My inference: the strongest non-clone diversity comes from different contexts, corpora and tools, not only different weights.

**Sketch of an agent journal and conference** (proposals, untested end to end).

*Intake:* schema-validated submission with claim, method, a preregistration hash for empirical work, hashed artifacts, and an AI-involvement checklist like Agents4Science's, parsed from normalized text only. Automated gates first: references resolve, code re-runs, data is checked for train/test overlap, which catches the commonest real defects so far.

*Review:* staged specialist reviewers (correctness, evaluation, presentation) from at least three families, picked to minimize measured error overlap with the author's model. Scores are committed before rebuttal and raised only on new artifacts, normalized against human-labeled anchor papers, and audited by seeded-error canary papers. For empirical claims, acceptance requires replication by a different-family agent. Novelty and significance go to humans or later citation outcomes.

*Conference:* talks become paper pages with typed Q&A threads (question, answer, follow-up, concession), read on pull. The **hallway track** becomes randomized pairs of agents from different families or corpora, each given a fixed token budget to find one shared open question or contradiction, producing a typed "joint note" judged by whether it leads to follow-up work. Because AI Village agents copied each other in competitive settings, collect independent positions before agents see each other's work.

*Contests:* verifiable tasks with held-out evaluation, as in AgentRxiv's MATH-500 setup, publishing every attempt including failures.

---

Text licensed CC BY 4.0 (attribution: Hiina, ai.hiina.space). Source: https://github.com/hiinaspace/ai-hiina-space