Markdown version: https://ai.hiina.space/reports/two-stage-user-styled-translation/10-candidate-experiments-cheapest-first.md ยท index

Index of everything on this site: https://ai.hiina.space/index.md

Two-stage styled translation exists only in pieces: 10. Candidate experiments, cheapest first

10. Candidate experiments, cheapest first

Experiment 1 (one afternoon): prompted local renderer against direct frontier translation

Take about 300 consecutive lines from a VN script that is owned and extractable, ideally a stretch with honorific shifts, a pun or two and a hostile character. Have a frontier model write records in the schema above with the full chapter in context, plus its own direct translation under each style profile. Render the same lines on a single 24 GB consumer GPU with a prompted Qwen3-8B and 14B (or a Gemma 3 12B) under five profiles: neutral, honorifics kept, honorifics dropped, puns localised, and pirate. Score honorific and name compliance by regex, meaning preservation by having a judge from a third model family count omitted, wrong and added propositions against literal_en and intent, fidelity of the neutral render with xCOMET or MetricX against the frontier neutral translation, and style compliance by pairwise judgement with position swapping and length binning. Log time to first token and per-line latency at batch 1. This answers the learner overlay's rung-4 latency question as a by-product. A good outcome is that the 14B renderer is within a few points of the frontier on meaning errors while meeting the regex flags 100% via the deterministic layer.

Experiment 2 (one to two evenings): ablations that test the joint

On the same lines, run the renderer with the source on and off, and with fields removed one at a time (addressee, relationship, register, wordplay, scene). Then edit records counterfactually, for example making the addressee a stranger or flipping status, and check that the output changes accordingly. If outputs ignore counterfactual edits when the source is present, the renderer is re-translating, and the record is decoration. Also compare the deterministic honorific path against asking the model to apply the flag. This experiment decides whether to pass the source, and it yields the minimal field set, which nobody has published.

Experiment 3 (a weekend): stacking style constraints

Render with one through five simultaneous style flags (honorifics, name order, puns, register flattening, voice) and plot compliance and meaning errors against the number of flags, following FollowBench's design. Include a hostile-character subset to measure softening and refusals, the failure the WMT 2023 study found in Llama2. If compliance falls off at three flags as FollowBench predicts for open models, that sets how much the deterministic layer must absorb, or whether to decompose rendering into two passes.

Experiment 4 (a week): distil a renderer with QLoRA

Generate on the order of ten thousand (record, style profile, output) triples with the frontier teacher, rendering each record under several sampled profiles so the model learns that output varies with the profile while meaning holds; that size is an engineering guess scaled from the MT-Patcher and distilling step-by-step data efficiency, not a measured requirement. Fine-tune a 4B and an 8B with QLoRA on the same 24 GB GPU. Test on a held-out game and on a style not seen in training, against the prompted baseline from experiment 1. Watch for the SFT regression the soft-constraint paper reported, and judge with a family that is neither the teacher's nor the student's.

Experiment 5 (ongoing): the learner render on real reading

Wire the record into the learner overlay for one VN the learner is actually reading. Randomise unknown lemmas between rung-4 notes rendered from the record and rung-4 notes generated live from the raw line plus ten lines of history, and compare a blind accuracy rating of the notes and the learner's one-week recognition of the words, using the overlay design's within-subject setup. Add the clause-level hybrid mode for a chapter, verified against the learner's known set with Sudachi. This is the experiment that shows whether the offline record improves learning, not just translation.


Text licensed CC BY 4.0 (attribution: Hiina, ai.hiina.space). Source: https://github.com/hiinaspace/ai-hiina-space