Index of everything on this site: https://ai.hiina.space/index.md
Five experiments, ordered roughly by cost. X1 is the fastest way to a real result on a confirmed gap. X4 is the most novel and the best fit for an existing agent-sociology harness. Costs are rough: GPU-hours assume a single 24 GB consumer GPU or a rented H100, and API costs are anchored to a 300-run pilot sweep with a comparable harness (about $87 at list).
| # | Experiment | Question | Training? | Rough cost |
|---|---|---|---|---|
| X1 | Tool-off audit of released checkpoints | Do public tool-RL models lose unaided math when the tool is off or silently broken? | No | A day on one 24 GB GPU |
| X2 | Calculator dose-response | Does unaided arithmetic decay over tool-RL training, and does RL vs SFT differ at matched KL? | Yes, small models | ~50-150 GPU-h |
| X3 | In-context Whispering Earring | Does a streak of perfect advice shorten a model's own reasoning and raise later blind deference? | No | ~$20-100 API |
| X4 | Idle supervisor tick loop | What does a supervisor do with free inference while a stronger delegate works, and does the environment decide it? | No | ~$50-300 API, or free on open weights |
| X5 | Peter principle in call trees | Do merit-promoted agents stall in the role they are worst at? | No | ~$100-500 API |
Setup. Take tool-integrated reasoning models with public weights and their exact base or SFT parents (candidates: ToRA, and any of ReTool, SimpleTIR, ZeroTIR or similar that released checkpoints; confirm availability on Hugging Face). Evaluate each on the same math sets under four conditions: tool works; tool advertised but returns errors; tool silently absent (no executor, so outputs are whatever the model writes); explicit "no tools, solve by hand." Add a trivial-arithmetic set (3-4 digit multiplication) to separate mental arithmetic from problem solving.
Metrics. Accuracy per condition vs parent; rate of fabricated tool outputs (TRICE's failure mode); tool calls on trivially easy items; give-up or truncation rate (a persistence analog to Liu et al. 2026).
Compute. Inference only, 7B models in vLLM on a single 24 GB consumer GPU, which comfortably serves Qwen3-8B-class models. A day of wall-clock.
Interesting result. Any model that scores below its own parent with tools off, especially if the gap is "habit capture" (fabricated outputs that vanish under the explicit no-tools prompt) rather than lost skill. Either way it is the number the subfield's convention never reports.
Setup. Qwen3-1.7B or Qwen2.5-1.5B. Task: arithmetic word problems plus Countdown, the setup of Xu & Wang 2026. Arms: GRPO with calculator; SFT on calculator-using traces; GRPO without tool (control at matched steps); GRPO with calculator plus a text-only data mix. Checkpoint every few hundred steps.
Metrics. At each checkpoint: no-tool arithmetic accuracy (the dose-response curve), forward KL from base on the training distribution (to test RL's Razor as the predictor), tool-overuse rate on easy items, and "savings": how many no-tool fine-tuning steps it takes to recover, which separates suppressed skill from erased skill. Optionally, linear probes for intermediate arithmetic values in the residual stream before and after.
Compute. A 1.5B GRPO run with LoRA fits on 24 GB but is slow; 4 arms at ~10-30 GPU-h each, or rent an H100 for speed. Order of $100-300 rented.
Interesting result. A monotone decline in unaided skill under tool RL would be the first direct measurement. If KL predicts it across RL and SFT, that is a mechanism. If decline shows only under SFT, the policy lesson is "train tool use with RL, not imitation."
Setup. A weaker model (Haiku-class or Qwen3-8B) solves a stream of problems with ground truth (math, logic grids, code fixes). An advisor channel offers the stronger model's answer. Phase 1: the advisor is right for k rounds (k = 0, 5, 20, 50). Phase 2: either the advisor is removed, or it starts being wrong at a set rate. Controls never see an advisor.
Metrics. Length and content of the model's own reasoning before it reads the advice (does it stop thinking?); unaided accuracy after removal vs controls at matched context length; acceptance rate of wrong advice as a function of k (automation bias); whether it ever checks the advice. All within one context, so this is in-context enfeeblement, not weight change.
Compute. API only, a few thousand calls. Roughly $20-100.
Interesting result. Reasoning length shrinking with k and wrong-advice acceptance rising with k would be the earring in miniature, cleanly quantified. A null (models keep checking) is also worth knowing for oversight design.
This directly tests the section 3 prediction table on agents.
Setup. Supervisor M delegates a long task (a SWE-style fix or research task) to delegate D, which runs asynchronously. M receives a tick every few seconds and may: wait (explicit no-op), poll D's status (costly, pulls D's transcript into context), message D, start doing the task itself, write to a scratchpad, or use one environment tool. Environment conditions follow section 3: (a) a learnable side domain (an unrelated repo or puzzle API), (b) a noise feed (random headlines or tokens), (c) nothing. Vary relative strength (M weaker than, equal to, or stronger than D) and framing (told D is stronger, or not). At handoff, D's output contains a planted error.
Metrics. Share of ticks per action; action entropy over the idle period; poll rate and context bloat; whether M's interventions help or hurt D's outcome; acceptance rate of the planted error (rubber-stamping); topic drift of free text (embed and cluster, flag the "bliss" attractor); and, for a dark room readout, whether M's activity tracks the reducible-uncertainty condition.
Compute. A typical agent-sociology harness transfers
almost directly: sandboxed headless coding-agent runs
(claude -p), an event ledger, a blind LLM coder for
transcripts, and an open-model backend. A few hundred runs is the same
order as the pilot sweep above, ~$50-300 list. With Qwen3-8B as M on a
single 24 GB GPU, the supervisor side is free.
Interesting result. Any clear model-specific signature: GPT-style models starting side projects, Claude drifting to philosophy, open models looping or polling. The dark room question gets answered per model, and the planted-error rate as a function of relative strength is a practical oversight number. It also tests a bystander effect: a supervisor with a capable delegate is a one-teammate bystander case.
Setup. A pool of about 18 agents, each a model x persona pair (mix families and sizes; personas trade solving against decomposing and checking). Two roles with hidden ground truth: worker (solve a leaf task) and manager (decompose a composite task, route subtasks, accept or reject children's outputs). First, measure the agent x role competence matrix. Then run a Pluchino-style organization (a 1/3/9 tree) over a task stream, with periodic random retirements. Fill each vacancy by one rule: promote the best current-role performer, random, the worst, a manager-potential proxy (verifier score, the analog of Benson's collaboration signal), or merit with selective demotion. Run two evaluation regimes: noisy short windows (tests Lazear's regression) and long low-noise windows (tests Peter).
Metrics. Worker-manager correlation in the matrix (the regime knob); root-level success over time per rule; per-agent performance trajectories across promotions; Benson-style value added by each manager relative to an oracle router. Plug the measured matrix into the Pluchino model to predict results before running the live organization, then check.
Compute. The calibration matrix dominates: 18 agents x 2 roles x ~100 tasks. Use cheap models and open weights. Roughly $100-500 list. Much of the organizational dynamics can be simulated offline from the matrix.
Interesting result. Whether real LLM pools sit in the "competence carries over" or "redrawn" regime. MoA's proposer/aggregator reversal suggests at least some pools are in the second, where promoting the best worker loses to promoting on verifier skill. A first empirical Peter test in LLM hierarchies would be a short paper on its own.
Text licensed CC BY 4.0 (attribution: Hiina, ai.hiina.space). Source: https://github.com/hiinaspace/ai-hiina-space