---
id: two-stage-user-styled-translation/05-a-small-renderer-is-plausible-measuring-it
title: 'Two-stage styled translation exists only in pieces: 5. A small renderer is
  plausible; measuring it honestly is the hard part'
report: https://ai.hiina.space/reports/two-stage-user-styled-translation.md
section: 5
version: 1
updated: '2026-10-27'
author: Hiina
---

Index of everything on this site: https://ai.hiina.space/index.md

# Two-stage styled translation exists only in pieces: 5. A small renderer is plausible; measuring it honestly is the hard part

- Section 5 of 12 of the report "Two-stage styled translation exists only in pieces" by Hiina (v1, updated 2026-10-27)
- Report overview and contents: https://ai.hiina.space/reports/two-stage-user-styled-translation.md
- Previous: https://ai.hiina.space/reports/two-stage-user-styled-translation/04-live-tools-are-single-stage-translators-with.md
- Next: https://ai.hiina.space/reports/two-stage-user-styled-translation/06-game-dialogue-systems-supply-the-field-vocabulary.md

## 5. A small renderer is plausible; measuring it honestly is the hard part

### Plan-then-realise and teacher annotations both favour the split

NLG has evidence that separating what to say from how to say it improves faithfulness. On WebNLG, a symbolic planner plus a neural realiser matched an end-to-end model on BLEU (47.4 vs 46.5) and cut omissions from 41 to 6, wrong lexicalisations from 39 to 17 and over-generation from 29 to 3 on 440 triplets. But the realiser covered the plan's entities in 98.9% of plans on seen data and only 66.7% on unseen data ([Moryossef et al. 2019](https://arxiv.org/pdf/1904.03396)). Expect a renderer to drift when a game's records look unlike its training data. A 2024 study of fine-tuned data-to-text found faithfulness tended to decline with model size while fluency rose, with T5-large matching Llama 2 on several metrics ([model-size study](https://arxiv.org/html/2407.14088v1), not citation-checked, no human eval).

Teacher-written intermediates train small students efficiently. A 770M T5 trained on PaLM 540B rationales beat few-shot PaLM on ANLI using 80% of the data, needed over 50% fewer examples on average than standard fine-tuning, and a weaker 20B teacher gave smaller gains ([Distilling step-by-step](https://arxiv.org/pdf/2305.02301)). MT-Patcher had a teacher write error spans, explanations and post-edits for a student's translations. A 104k feedback-selected set matched or beat 1M-sentence sequence-level distillation for NLLB 3.3B zh-en (COMET 80.3 vs 79.5), and the en-ja student rose from 85.1 to 86.4 COMET with a LLaMA2 teacher ([MT-Patcher, NAACL 2024](https://arxiv.org/pdf/2403.09522)). Unlike rationale distillation, the record is needed at inference, so the gain is a cost shift offline, not a teacher-free student.

### Small models follow two or three constraints reliably, not five

Modern small instruct models score well on verifiable English instructions: Qwen3 4B, 8B and 14B reach 81.2, 83.0 and 84.8 IFEval strict-prompt in non-thinking mode ([Qwen3 report](https://arxiv.org/pdf/2505.09388)). Gemma 3 self-reports higher IFEval under a different scoring setup and a translation score (WMT24++, a chrF-type character F-score) of 46.8 at 4B and 51.6 at 12B ([Gemma 3 report](https://arxiv.org/pdf/2503.19786)); its IFEval should not be compared to Qwen's table. Stacked constraints degrade: on FollowBench, GPT-4's hard satisfaction fell from 84.7% at one constraint to 61.9% at five, and Vicuna-7B from 60.8% to 23.9%; style was GPT-4's easiest category (92.7%), and the authors estimate closed models satisfy about three constraints and open models about two ([FollowBench](https://arxiv.org/pdf/2310.20410), 2023-era models). One follow-up found plain SFT slightly lowered LLaMA3-8B's FollowBench score while judge-guided DPO raised it ([soft-constraint paper](https://arxiv.org/pdf/2501.04945), not citation-checked). A style profile of honorifics, puns, register and voice is four or more constraints on top of fidelity, which is why the deterministic layer should take the closed-class flags off the model's plate.

Persona drift is a design constraint, not a blocker. LLaMA2-chat-70B drifted from its system-prompt persona within eight dialog rounds, with gpt-3.5-turbo-16k also drifting, and the authors' split-softmax fix amplifies attention to the system prompt ([Li et al., Measuring and Controlling Persona Drift](https://arxiv.org/html/2402.10962v1)). A renderer that makes each line an independent call with the style profile freshly at the end of the prompt, as Character Card V2's `post_history_instructions` does ([Character Card V2 spec](https://github.com/malfoyslastname/character-card-spec-v2/blob/main/spec_v2.md)), avoids accumulation. Persona vectors steer traits such as sycophancy and humour in 7-8B open models at some cost to capability, but were not shown for speaking style ([Anthropic, Persona vectors](https://www.anthropic.com/research/persona-vectors), not citation-checked).

### Latency fits on one consumer GPU; evaluation needs targeted checks and a cross-family judge

A third-party llama.cpp benchmark on an RTX 4090 (24 GB; Q4_K_XL, flash attention, model family not named) measured 8B generation at 131 tokens/s and prompt processing at 9,121 tokens/s at 4K context, and 14B at 82.8 and 5,358 ([Hardware Corner](https://www.hardware-corner.net/rtx-4090-llm-benchmarks/), secondary, not peer-reviewed). From those figures (my arithmetic, not a measurement), a 600-token prompt and 60-token line costs about 0.5-0.6 s on 8B and 0.9 s on 14B cold, and prefix caching of the shared system prompt cuts prefill further ([vLLM prefix caching](https://docs.vllm.ai/en/v0.8.5/features/automatic_prefix_caching.html), listing only). Grammar-constrained JSON decoding costs under 40 microseconds per token mask with XGrammar ([XGrammar](https://arxiv.org/html/2411.15100v3)), which matters for the offline record writer, not the plain-text renderer. UGTLive's author guesses 3-6 s for a whole live OCR-plus-LLM loop ([UGTLive](https://github.com/SethRobinson/UGTLive)), so pre-rendering ahead is what makes any of this feel instant.

Evaluation is where naive setups mislead. Neural QE metrics and LLM judges penalise length: appending one to five error-free segments dropped scores by 4.0-8.0 points for MetricX-24 QE, 1.5-5.6 for CometKiwi, 1.7-5.3 for Gemini AutoMQM and 1.8-8.5 for Gemini as a judge, across about 80% of documents, and MetricX and CometKiwi preferred the shorter of two comparable translations 50.4-60.4% of the time ([Penalizing Length](https://arxiv.org/html/2510.22028v3)). A pirate render is longer and will be marked down for it. Judges prefer their own family, and small judges more so (Llama-3.1-8B bias 21.6% against 0.4% for 70B; Qwen2.5 0.5B 41.7% against 2.1% for 14B) ([Beyond the Surface](https://arxiv.org/pdf/2506.02592)), so the judge must come from neither the teacher's nor the renderer's family. For style accuracy GPT-4 tracked humans best (Pearson 0.92 on English sentiment transfer), but for content preservation BERTScore and BLEURT matched or beat it in several cells ([Evaluating TST evaluation](https://arxiv.org/pdf/2502.04718)), so no single judge covers both. The WebNLG error counts (omitted, wrong, added) are the right shape for checking a styled render against the record's meaning field.

---

Text licensed CC BY 4.0 (attribution: Hiina, ai.hiina.space). Source: https://github.com/hiinaspace/ai-hiina-space