---
id: ai-enfeeblement-and-the-peter-principle/02-ai-enfeeblement-as-measured
title: 'AI enfeeblement, idle supervisors and the Peter principle in agents: 2. AI
  enfeeblement as measured'
report: https://ai.hiina.space/reports/ai-enfeeblement-and-the-peter-principle.md
section: 2
version: 1
updated: '2026-10-27'
author: Hiina
---

Index of everything on this site: https://ai.hiina.space/index.md

# AI enfeeblement, idle supervisors and the Peter principle in agents: 2. AI enfeeblement as measured

- Section 2 of 9 of the report "AI enfeeblement, idle supervisors and the Peter principle in agents" by Hiina (v1, updated 2026-10-27)
- Report overview and contents: https://ai.hiina.space/reports/ai-enfeeblement-and-the-peter-principle.md
- Previous: https://ai.hiina.space/reports/ai-enfeeblement-and-the-peter-principle/01-summary.md
- Next: https://ai.hiina.space/reports/ai-enfeeblement-and-the-peter-principle/03-the-dark-room-problem-and-offloading.md

## 2. AI enfeeblement as measured

The exact experiment the question describes (train with a calculator, then test mental arithmetic without it, across training) has not been run. That gap held across about eight search wordings. The closest direct evidence says well-designed tool training preserves unaided ability, while supervised fine-tuning on tool-only traces can destroy it. Behavioral over-reliance, where models call tools they don't need, is well documented. On the human side, deskilling is real in high-stakes automation and in "practice with AI, then test without" trials, while several famous pop results are weak.

### Models: tool on vs tool off

| Study | Setup | Unaided ability after tool training |
| --- | --- | --- |
| [Toolformer, Schick 2023](https://arxiv.org/pdf/2302.04761) | GPT-J 6B, self-supervised tool fine-tune incl. calculator; evaluated with API calls disabled | No harm, slight gain (ASDiv 7.5 to 14.8); near floor, small model |
| [TRICE, Cheng 2026](https://arxiv.org/pdf/2605.06326) | Qwen3 4B/30B thinking models, code interpreter | Final recipe keeps or improves text-only math. Ablation fine-tuned only on tool traces: text-only BeyondAIME 54.3 to 34.1, with the model writing code, inventing its output and looping. Mixing text-only traces back recovers most of it |
| [CITI, Hao 2024](https://arxiv.org/html/2409.13202v2) | Full fine-tune of Llama-3-8B on API-Bank | Large collateral loss (GSM8K 79.5 to 26.4, HumanEval 59.2 to 0); LoRA avoids it. Ordinary catastrophic forgetting, not skill substitution |
| [Xu & Wang 2026](https://arxiv.org/html/2608.28447v1) | RL on Countdown arithmetic with a calculator | No tool-off eval of the tool-trained model. The cheapest place to run the missing experiment |

TRICE's failure mode is the most interesting data point. The model didn't just get worse; it kept the tool-shaped reasoning loop and hallucinated the tool's output when the tool was gone. That is enfeeblement as habit capture, not only skill decay.

Most tool-integrated reasoning papers ([ReTool](https://arxiv.org/pdf/2504.11536), [SimpleTIR](https://arxiv.org/pdf/2509.02479), [Lin & Xu 2025](https://arxiv.org/pdf/2508.19201)) compare against a separately trained text-only model, so the "same model, tool switched off" number is almost never reported. That convention is why the gap exists. Search-RL papers likewise never report closed-book recall for the trained model.

### Models: over-reliance as behavior

- [SMART, Qian 2025](https://arxiv.org/html/2502.11435v1): models call tools unnecessarily over 30% of the time; Llama-3.1-8B 66% on GSM8K.
- [Tool-Overuse Illusion, Zeng 2026](https://arxiv.org/html/2604.19749v1): about 0.93 unnecessary calls per simple GSM8K problem, tool use lowers accuracy on easy problems, and RL with verifiable rewards raises tool turns during training (Qwen2.5-7B 2.2 to 6.8).
- [Tool-Induced Myopia, Bayat 2025](https://arxiv.org/html/2511.10899v1): with a code tool, final-answer accuracy rises but reasoning quality falls, as brute-force checks replace derivation.
- [Paturi 2026](https://arxiv.org/abs/2609.14157): merely making a related but unnecessary tool available drops the rate of answering closed questions from 98.2% to 63.5% (abstract only).
- [OTC-PO, Wang 2025](https://arxiv.org/html/2504.14870v2): penalizing tool calls cuts Search-R1 calls from 3.28 to 1.04 per question with accuracy unchanged, so the extra calls were pure habit.

### Models: adjacent mechanisms

- [RL's Razor, Shenfeld 2025](https://arxiv.org/html/2509.04259v1): at equal new-task performance, RL forgets less than SFT, and forgetting tracks forward KL from the base model. This is the likely explanation for TRICE's RL-vs-SFT contrast and a testable prediction.
- [Yue et al. 2025](https://arxiv.org/pdf/2504.13837): RLVR raises pass@1 but shrinks pass@256 below the base model. A narrowing rather than enfeeblement.
- [Lin et al. 2024](https://arxiv.org/html/2309.06256v4): the RLHF alignment tax on translation and reading comprehension; weight averaging recovers much of it.
- [Mallen et al. 2023](https://arxiv.org/html/2212.10511v4): retrieval rescues about 17% of PopQA answers but breaks about 10% the model already knew. The original case for using a tool only when the model's own knowledge is weak.

### Humans

| Finding | Design | Confidence |
| --- | --- | --- |
| [Bastani 2025, PNAS](https://ideas.repec.org/a/nas/journl/v122y2025pe2422633122.html): plain GPT-4 practice raised practice grades 48% but exam scores without AI fell 17%; a tutor version avoided the harm | Field RCT, \~1,000 students | High |
| [Shen & Tamkin 2026, Anthropic](https://www.anthropic.com/research/AI-assistance-coding-skills): engineers learning a library scored 50% with AI vs 67% without on a later quiz; delegators did worst | RCT, n=52 | Medium |
| [Liu et al. 2026](https://arxiv.org/abs/2604.04721): brief AI help lowered later unaided performance and persistence | RCTs, n=1,222 (abstract only) | Medium |
| [Maier et al. 2026](https://arxiv.org/html/2609.20143v1): AI access did not significantly change the later unaided fractions test (OR 0.80, 95% CI 0.42-1.51, so harm is not ruled out); feedback on reliance helped | Preregistered, n=704 | Medium, partly null |
| [Budzyń 2025, Lancet Gastro](https://wrap.warwick.ac.uk/id/eprint/191005): adenoma detection without AI fell 28.4% to 22.4% after AI was introduced | Observational, 1,443 procedures | Medium |
| [Casner 2014](https://gwern.net/doc/technology/2014-casner.pdf): 747 pilots kept hand-flying skills but lost cognitive navigation skills; mind-wandering 20% of thought with automation on vs 5% off, correlated with errors | Simulator, n=16 | Medium |
| [Parasuraman & Manzey 2010](https://trid.trb.org/View/1085175): automation complacency and bias affect experts and survive training | Review | High |
| [Hembree & Dessart 1986](https://concept.journals.villanova.edu/article/viewFile/139/110): calculators did not hurt paper skills in K-12 except sustained use in grade 4 | Meta-analysis, 79 studies (secondary summary) | Medium |
| [Dahmani & Bohbot 2020](https://pmc.ncbi.nlm.nih.gov/articles/PMC7156656/): more GPS use predicted steeper spatial memory decline | Longitudinal n=13 | Low |
| [Woollett & Maguire 2011](https://www.fil.ion.ucl.ac.uk/Maguire/WoollettMaguireCBDec2011.pdf): taxi trainees who qualified grew posterior hippocampus | Longitudinal, self-selected | Medium (use-it side) |
| Sparrow 2011 "Google effect": [failed replication](https://peerj.com/articles/10325) (BF 5 for null) and in [Camerer 2018](https://forrt.org/flora-replication-atlas/doi/10.1126/science.1207745/) | Replications | Treat as unreliable |
| [Kosmyna 2025, "Your Brain on ChatGPT"](https://arxiv.org/abs/2506.08872v1): lower EEG connectivity with LLM use | n=54, 18 in crossover; [critiqued](https://arxiv.org/abs/2601.00856v1) | Low |
| [Lee et al. CHI 2025](https://advait.org/files/lee_2025_ai_critical_thinking_survey.pdf), Gerlich 2025: AI confidence correlates with less critical thinking | Self-report surveys | Low for causation |

The human gap: no multi-week study randomizes LLM access and then tests unaided skill after a washout. [Chooi, Lee & Li 2026](https://aclanthology.org/2026.evaleval-1.36/) make the matching point for AI evals: disempowerment metrics measure exposure, not skill erosion.

Casner's pilot result is the best existing answer to the "what does the human do while the AI works" question: the freed attention goes to mind-wandering, and that predicts later failure. That points toward quiescence, not productive substitution.

---

Text licensed CC BY 4.0 (attribution: Hiina, ai.hiina.space). Source: https://github.com/hiinaspace/ai-hiina-space