---
id: ai-enfeeblement-and-the-peter-principle/03-the-dark-room-problem-and-offloading
title: 'AI enfeeblement, idle supervisors and the Peter principle in agents: 3. The
  dark room problem and offloading'
report: https://ai.hiina.space/reports/ai-enfeeblement-and-the-peter-principle.md
section: 3
version: 1
updated: '2026-10-27'
author: Hiina
---

Index of everything on this site: https://ai.hiina.space/index.md

# AI enfeeblement, idle supervisors and the Peter principle in agents: 3. The dark room problem and offloading

- Section 3 of 9 of the report "AI enfeeblement, idle supervisors and the Peter principle in agents" by Hiina (v1, updated 2026-10-27)
- Report overview and contents: https://ai.hiina.space/reports/ai-enfeeblement-and-the-peter-principle.md
- Previous: https://ai.hiina.space/reports/ai-enfeeblement-and-the-peter-principle/02-ai-enfeeblement-as-measured.md
- Next: https://ai.hiina.space/reports/ai-enfeeblement-and-the-peter-principle/04-idle-supervisor-agents.md

## 3. The dark room problem and offloading

The dark room literature does not, on its own, predict whether an agent relieved of its task goes quiet or finds something else to do. The outcome depends on two things an experiment can control: what the agent's model expects of itself, and whether any uncertainty is left that its own actions can reduce. Human boredom data point toward substitution, but often onto low-value or aversive stimuli, while the offloaded skill still decays. I found no published work connecting the dark room problem to AI delegation or enfeeblement; that gap looks real.

### The debate in brief

- **The problem.** If organisms minimize surprise, why not sit in a dark room forever?
- **Friston's answer** ([Friston, Thornton & Clark 2012](https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2012.00130/full)): surprise is relative to a model. Phenotype and niche set priors under which a dark room is surprising (you expect to be fed, warm, moving). Clark already worries the framework is "too accommodating."
- **Curiosity is built in, not derived.** Expected free energy splits into pragmatic value and epistemic value (information gain). [Schwartenbeck et al. 2013](https://discovery-pp.ucl.ac.uk/1423327/1/fpsyg-04-00710.pdf): when outcome utilities are flat, policies that maximize entropy over outcomes win, weighted by precision. But [Millidge, Tschantz & Buckley 2020](https://arxiv.org/pdf/2004.08128) show the natural future extension of variational free energy carries information gain with the opposite sign. A "pure" free-energy agent leans toward the dark room; exploration comes from a modeling choice.
- **The critique** ([Sun & Firestone 2020](https://perception.jhu.edu/files/PDFs/20_DarkRoom/SunFirestone_DarkRoom_TiCS.pdf), [reply](https://perception.jhu.edu/files/PDFs/20_DarkRoom/SunFirestone_DarkRoomReply_TiCS_InPress.pdf)): any prediction error can be cancelled by revising the prediction instead of acting, and the self-prediction reply ("we don't expect to be in dark rooms") is true but accommodates any behavior. The commentaries by [Van de Cruys, Friston & Clark](https://researchers.mq.edu.au/en/publications/controlled-optimism-reply-to-sun-and-firestone-on-the-dark-room-p/) and [Seth et al.](https://sro.sussex.ac.uk/id/eprint/91163/) appeal to optimism priors and long planning horizons (I could only read Sun & Firestone's summary of them).
- **The most useful variant for enfeeblement:** [Kiverstein, Miller & Rietveld 2019](https://pure.uva.nl/ws/files/37405887/Kiverstein2019_Article_TheFeelingOfGripNoveltyErrorDy.pdf) argue valence tracks the *rate* at which error falls. Agents seek "progress niches" slightly beyond their skill (compare Oudeyer's learning progress). [Sims](https://predictive-mind.net/epubs/the-problems-with-prediction/OEBPS/23_Sims.xhtml) concludes the framework handles "why not stay in the room" but the triviality worry stands.

**Testable?** Not in general. It becomes testable once the generative model is fixed: priors, planning horizon, and which objective. Schwartenbeck's flat-utility condition is the cleanest conditional prediction.

### Human evidence: substitution, often badly aimed

- [Wilson et al. 2014](https://www.simplypsychology.org/?p=56986): left alone 6-15 minutes, 67% of men and 25% of women shocked themselves rather than just think. [Fox et al. 2014](https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2014.01427/full) note mean enjoyment was mildly positive and many shockers cited curiosity, so it reads as novelty seeking more than aversion to thought.
- [Bench & Lench](https://www.psypost.org/boredom-drives-people-to-seek-new-experiences-even-when-theyre-unpleasant/): bored people choose novel unpleasant images over familiar neutral ones.
- [Trudel et al. 2025](https://repository.essex.ac.uk/40202/1/Boredom%20signals%20deviation%20from%20a%20cognitive%20homeostatic%20set%20point.pdf): boredom signals deviation from an engagement set point and prompts a novelty search that can be maladaptive. That set point is close to the phenotype prior the dark room replies need. [Danckert et al. 2018](https://cs.uwaterloo.ca/~jhoey/teaching/cogsci600/papers/Danckert2018.pdf) frame it as an opportunity-cost signal.
- [Eisenberger 1992](https://homepages.se.edu/cvonbergen/files/2013/01/Learned-Industriousness.pdf) (learned industriousness) and the [effort paradox](https://mindblog.dericbownds.net/2018/05/the-effort-paradox-effort-is-both.html): effort that pays off acquires value of its own. My inference: removing occasions where effort pays should erode that over time.

### RL analogs

- [SMiRL, Berseth et al. 2021](https://ar5iv.arxiv.org/html/1912.05510) is literally a dark room agent (reward = log-probability of its own state). In unstable environments it acts purposefully (clears Tetris lines, dodges fireballs); in static ones it finds trivial low-entropy policies. Activity comes from the environment's entropy, not the objective.
- [Curiosity-driven learning, Burda 2018](https://arxiv.org/abs/1808.04355) and the [noisy TV problem](https://arxiv.org/html/2102.04399v3): prediction-error curiosity fixates on irreducible noise. This is the RL version of humans picking the shock button.
- [Empowerment](https://arxiv.org/pdf/1406.1767): agents maximizing action-to-future-state channel capacity reshape their world to keep options open.

### The Whispering Earring

In [Scott Alexander's 2012 story](https://gwern.net/doc/fiction/science-fiction/2012-10-03-yvain-thewhisperingearring.html), the earring first says "Better for you if you take me off," then always gives advice better than the wearer's own choice, eventually down to individual muscle contractions. Wearers live successful lives; autopsies show a wasted neocortex and enlarged reflexive brain regions. In dark room terms, the wearer is never idle: policy selection is taken over and the wearer becomes the executor. That is a third outcome beside "rot" and "find something else." Recent LessWrong takes ([Dentosal 2026](https://www.greaterwrong.com/posts/cQkSh9b48WbTaiu2a/sadly-the-whispering-earring), [AlphaAndOmega 2026](https://www.greaterwrong.com/posts/t5psGMdgReuRucEkr/contra-the-usual-interpretation-of-the-whispering-earring)) argue whether it is a prosthesis or atrophy. AI-risk framings ([Hendrycks et al. 2023](https://arxiv.org/pdf/2306.12001) on enfeeblement; [Gradual Disempowerment, Kulveit et al. 2025](https://arxiv.org/abs/2501.16946)) work at the level of society, not cognition. Two 2026 atrophy papers ([Trajanov & Kocarev](https://www.preprints.org/manuscript/202606.1628), [Mandalapu et al.](https://aisel.aisnet.org/amcis2026/ai_systdesign/ai_systdesign/21/)) never mention the dark room.

### What it predicts when a stronger helper removes the task

This part is synthesis, mostly speculative. A helper that guarantees success flattens pragmatic value across the agent's own policies. Different objectives then diverge:

| Agent objective | Learnable side-domain available | Only a noise source | Nothing else |
| --- | --- | --- | --- |
| Expected free energy with information gain | Engages it | Ignores it | Quiescent or habitual |
| Prediction-error curiosity | Engages it | Captured by it | Wanders |
| Surprise minimization (SMiRL) | Quiescent | Quiescent | Quiescent |
| Empowerment | Works on controlling the helper | Same | Same |
| Humans (boredom data) | Plausibly engages | Plausibly captured | Aversive restlessness |

The deciding variable is probably not whether the agent has a task but whether it has **uncertainty it can reduce through its own actions**. If the helper also explains, plans and monitors, the epistemic term collapses too, and only habit is left. Kiverstein's account suggests "brain rot" and "finding something else" are not alternatives: expect specific atrophy in the offloaded domain plus substitution elsewhere. For an LLM, the "phenotype prior" is its pre- and post-training distribution, which is why section 4's observed defaults differ by model. Experiment X4 below operationalizes this table.

---

Text licensed CC BY 4.0 (attribution: Hiina, ai.hiina.space). Source: https://github.com/hiinaspace/ai-hiina-space