--- id: story-decompiling-and-remixing/08-five-experiments-cheapest-first title: 'Story decompilers exist, but nobody closes the loop: 8. Five experiments, cheapest first' report: https://ai.hiina.space/reports/story-decompiling-and-remixing.md section: 8 version: 1 updated: '2026-10-27' author: Hiina --- Index of everything on this site: https://ai.hiina.space/index.md # Story decompilers exist, but nobody closes the loop: 8. Five experiments, cheapest first - Section 8 of 10 of the report "Story decompilers exist, but nobody closes the loop" by Hiina (v1, updated 2026-10-27) - Report overview and contents: https://ai.hiina.space/reports/story-decompiling-and-remixing.md - Previous: https://ai.hiina.space/reports/story-decompiling-and-remixing/07-edits-that-stay-coherent-are-local-typed.md - Next: https://ai.hiina.space/reports/story-decompiling-and-remixing/09-conclusion.md ## 8. Five experiments, cheapest first The experiments below share one corpus and one scorer, so each later step reuses the earlier ones. The cost estimates are our own rough arithmetic, not sourced figures. ### Experiment 1. Schema rate-distortion on short unseen stories (API only, tens of dollars) Take 50 to 100 short stories (1,000 to 3,000 words) that no model has seen: stories written for the experiment, post-cutoff publications that can legally be used, or pseudonymised and perturbed Gutenberg tales such as FlawedFictions' originals. Compress each to five levels: logline; ATU, Propp and trope tags only; beat sheet; scene cards plus bible; and a prose synopsis at the same token count as the scene cards. Expand each back with a model from a different family than the extractor, to blunt self-preference. Score each reconstruction three ways. First, claim recall and precision: extract atomic claims from source and reconstruction, then check each against the other with whole-text context, FABLES-style. Second, answer source-derived questions from the reconstruction alone. Third, run a SemEval-style pairwise same-story judgement against genre-matched decoys after pseudonymisation. Run the decoy set (shuffled schema, negated-fact schema, other-story schema) through the same scorer and report the false-pass rate, as with any shifted-control checker. The output is the first plot rate-distortion curve. It answers whether any structured schema beats a prose synopsis at equal budget. Add Hamlet, Cinderella and two or three other canonical works, both raw and pseudonymised, as a memorisation arm. The gap between their fidelity and the unseen stories' fidelity measures how much the decoder's prior does the work. ### Experiment 2. Single typed edits, schema-side against prompt-side (API only, tens of dollars) On the same stories, apply one edit at a time: a setting swap, one character's gender, one Propp function (the villain's motive, or the ending's outcome), and one negated setup fact. Do each edit two ways. In one, edit the schema and expand. In the other, ask a strong model to rewrite the full original text with the same instruction ("rewrite this story so that..."). Score three things: edit adherence (targeted attribute QA on the output), invariance (claim recall outside the edit's scope after neutralising the edited attribute, a DTO-style delta against the original), and new contradictions (a FlawedFictions-style plot-hole check plus a scene-card consistency ledger). This is the experiment that tests the core hypothesis, that a mid-level representation makes edits more precise than prompting. The literature predicts the prompt-side baseline will over-edit and add plot holes. The open question is whether schema-side edits leave stale dependencies, such as an unchanged succession plot after a gender swap, that the expander papers over. ### Experiment 3. Stacked edits and the deterministic layer (API, under a hundred dollars) Apply one to four simultaneous edits (setting, gender, genre, era) and plot adherence, invariance and contradiction rates against edit count, following FollowBench's design. Run every condition twice: once with closed-class edits (names, pronouns, setting nouns) applied in code to the bible before expansion, and once with every edit given to the LLM. If the curves split at two or three edits, as FollowBench predicts for open models, that sets how much the deterministic layer must absorb. A Lovecraftian, gender-swapped Hamlet is the showcase case for this run, after the pseudonymisation control in experiment 1 has measured memorisation. ### Experiment 4. A local expander on a single 24 GB consumer GPU (one to two days of GPU time) Fine-tune a 7 to 8B model (Qwen-class, LoRA) as the expander. Train on (scene cards plus bible, story) pairs from experiment 1's extractor run over a few thousand public-domain or licensed stories. Mix in forward-synthesised pairs, which Lei et al. found fixes the input-output misalignment of reverse-only data. Compare its round-trip curve with the API expander's. SuperWriter's distilled 7B came within 0.04 of DeepSeek-R1's overall WritingBench score (8.51 against 8.55), which suggests the gap may be small ([SuperWriter](https://arxiv.org/pdf/2506.04180v1)). A local expander makes the large sampling sweeps of experiments 2 and 3 nearly free. If the extractor is later optimised against round-trip loss, add the NLA-style paraphrase test on the schema to rule out side channels. ### Experiment 5. Screenplay-scale round trip, then film (a few dollars per film plus human time) Move up to screenplays. MovieSum supplies 2,200 cleanly formatted screenplays paired with Wikipedia plots ([MovieSum](https://arxiv.org/html/2408.06281v1)). Decompile each screenplay to scene cards in chunks, regenerate a screenplay, and score it against the real one. Use scene alignment, dialogue overlap for the scenes that survive, the claim scorer from experiment 1, and ScreenWriter's protocol of checking that a summary of the regeneration agrees with the Wikipedia plot. Use films released after the models' cutoffs where possible, and treat famous films as memorisation controls. Only then add the video path: per-scene Gemini ingestion of legally owned films, aligned to subtitles and compared with the real screenplay. That gives the first whole-film video-to-script evaluation against true screenplays that we found. Expect character naming and scene boundaries to be the weak fields, as in ScreenWriter and OmniScript. Plan for screenplay rights and the films' own licences to limit what can be published, even where the experiment itself is cheap. --- Text licensed CC BY 4.0 (attribution: Hiina, ai.hiina.space). Source: https://github.com/hiinaspace/ai-hiina-space