---
id: story-decompiling-and-remixing/06-round-trip-fidelity-needs-decoys-because-the
title: 'Story decompilers exist, but nobody closes the loop: 6. Round-trip fidelity
  needs decoys because the decoder already knows Hamlet'
report: https://ai.hiina.space/reports/story-decompiling-and-remixing.md
section: 6
version: 1
updated: '2026-10-27'
author: Hiina
---

Index of everything on this site: https://ai.hiina.space/index.md

# Story decompilers exist, but nobody closes the loop: 6. Round-trip fidelity needs decoys because the decoder already knows Hamlet

- Section 6 of 10 of the report "Story decompilers exist, but nobody closes the loop" by Hiina (v1, updated 2026-10-27)
- Report overview and contents: https://ai.hiina.space/reports/story-decompiling-and-remixing.md
- Previous: https://ai.hiina.space/reports/story-decompiling-and-remixing/05-the-near-misses-each-stop-one-step.md
- Next: https://ai.hiina.space/reports/story-decompiling-and-remixing/07-edits-that-stay-coherent-are-local-typed.md

## 6. Round-trip fidelity needs decoys because the decoder already knows Hamlet

The autoencoder framing has one trap that outweighs every other design question: **a strong decoder's prior does much of the reconstruction**. The in-context autoencoder (ICAE) reconstructs ordinary 500-token passages near-verbatim from 128 memory slots, but random text scores BLEU 3.5 or 0.2 against 99.3 for normal text ([ICAE](https://arxiv.org/pdf/2307.06945)). For stories, a decoder handed "Danish prince, ghost, revenge, poison" will write Hamlet from memory, and the round trip will look nearly lossless with a near-empty schema. The Propp tagging result above, where few-shot examples did not help ChatGPT, shows the same prior on the extraction side. Fidelity experiments therefore need stories the model has not seen: newly written fiction, post-cutoff publications, or pseudonymised and perturbed known works. Canonical works belong in the experiment as a measured memorisation control, not as the test set.

The second trap is a schema that games the metric. Anthropic's Natural Language Autoencoders turn activations into text and back, trained only on reconstruction error. They tested for hidden side channels with meaning-preserving rewrites of the intermediate text (shuffling, paraphrase, translation into French), which caused only small drops in reconstruction quality. They also found the reconstructor a weak per-claim verifier, although removing true claims hurt reconstruction more than removing false ones ([NLA](https://transformer-circuits.pub/2026/nla/)). If a story extractor is ever optimised against round-trip loss, the same paraphrase test is the cheap check that the schema is carrying meaning and not a code.

### No single judge will do

Content-fidelity metrics on narrative are weak, so no single judge will do. On FABLES (26 books from 2023 to 2024, 3,158 annotated claims), the best automatic rater, Claude-3-Opus given the whole book, found unfaithful claims at only **F1 58.2**. Retrieving evidence with BM25 scored 24.9. Omissions appeared in 52 to 85% of summaries, and chronology errors in a third to more than half ([FABLES](https://arxiv.org/html/2404.01261v2)). On STORYSUMM (32 Reddit stories, 96 summaries), no automatic method passed 70% balanced accuracy. Even human labelling protocols disagreed sharply: three experts agreed on only 46% of labels before adjudication ([STORYSUMM](https://arxiv.org/pdf/2407.06501)). LLM judges increasingly prefer LLM-written summaries as overlap with the human reference drops ([Blind to the Human Touch](https://arxiv.org/html/2602.07673v1)). In TTCW, LLM judges' agreement with expert creativity ratings was essentially zero (Cohen's kappa about 0.0) ([TTCW](https://arxiv.org/pdf/2309.14556)). FABLES' error types (character states, events, causal links) are exactly what a lossy round trip produces, and precision-style faithfulness checks miss omissions entirely. Both directions are needed: recall of source claims in the reconstruction, and support of reconstruction claims by the source.

### A "same story" benchmark, and its shortcuts

A "same story" test now exists as a benchmark. SemEval-2026 Task 4 asks which of two candidate synopses is more similar to an anchor. Similarity is defined over abstract theme, course of action and outcomes, and explicitly excludes names and settings. That definition is the invariant wanted under a setting swap. The best systems reached **78% (pairwise LLM track) and 72% (embedding track)**, against an estimated label ceiling of about 89%. Accuracy correlated with genre clustering in embedding space at r = 0.67 ([SemEval-2026 Task 4](https://www.inf.uni-hamburg.de/en/inst/ab/lt/publications/2026-hatzel-et-al-semeval.pdf)). Embeddings partly track genre, not plot, so decoys must be genre-matched. Names are the other shortcut. On unmasked text a general sentence encoder ties a story-specific embedding model at about 95% P@1, but on pseudonymised text the gap opens to 82.6 against 67.3 ([Story Embeddings](https://aclanthology.org/2024.emnlp-main.339.pdf)). Pseudonymise before comparing, or the metric rewards preserved names. That would make "Hamlet in space with new names" look like a different story, and a name-swapped copy look like a new one.

### Budget-matched baselines and decoys

The two remaining design rules come from compression research. ICAE compared its 128 memory slots against a GPT-4 summary of the same 128-token budget, and the slots won 34.1% to 17.6% with 48.3% ties ([ICAE](https://arxiv.org/pdf/2307.06945)). The story analogue is a **plain prose synopsis at a matched token budget**: any structured schema has to beat it, or structure is getting credit for nothing. The human reference point for compression is the Wikipedia plot. MovieSum pairs screenplays of about 29K words with plots of about 714 to 717 words, roughly 40:1 ([MovieSum](https://arxiv.org/pdf/2408.06281)). We found no plot rate-distortion curve in the literature. Building one, with fidelity against tokens for each schema level, is itself a contribution.

The decoy design is the shifted-control pattern familiar from other LLM claim checkers. Score the true source-against-reconstruction pair alongside reconstructions from a same-genre different story, from the source schema with events shuffled, from the schema with one setup fact negated, and from a pseudonymised source. Report the checker's false-pass rate on the decoys next to its pass rate on true pairs. The negated-fact decoy has a ready-made generator in FlawedFictions, which builds plot holes by extracting a first-act fact and contradicting it later ([FlawedFictions](https://arxiv.org/html/2504.11900v3)).

---

Text licensed CC BY 4.0 (attribution: Hiina, ai.hiina.space). Source: https://github.com/hiinaspace/ai-hiina-space