Index of everything on this site: https://ai.hiina.space/index.md
NLG has evidence that separating what to say from how to say it improves faithfulness. On WebNLG, a symbolic planner plus a neural realiser matched an end-to-end model on BLEU (47.4 vs 46.5) and cut omissions from 41 to 6, wrong lexicalisations from 39 to 17 and over-generation from 29 to 3 on 440 triplets. But the realiser covered the plan's entities in 98.9% of plans on seen data and only 66.7% on unseen data (Moryossef et al. 2019). Expect a renderer to drift when a game's records look unlike its training data. A 2024 study of fine-tuned data-to-text found faithfulness tended to decline with model size while fluency rose, with T5-large matching Llama 2 on several metrics (model-size study, not citation-checked, no human eval).
Teacher-written intermediates train small students efficiently. A 770M T5 trained on PaLM 540B rationales beat few-shot PaLM on ANLI using 80% of the data, needed over 50% fewer examples on average than standard fine-tuning, and a weaker 20B teacher gave smaller gains (Distilling step-by-step). MT-Patcher had a teacher write error spans, explanations and post-edits for a student's translations. A 104k feedback-selected set matched or beat 1M-sentence sequence-level distillation for NLLB 3.3B zh-en (COMET 80.3 vs 79.5), and the en-ja student rose from 85.1 to 86.4 COMET with a LLaMA2 teacher (MT-Patcher, NAACL 2024). Unlike rationale distillation, the record is needed at inference, so the gain is a cost shift offline, not a teacher-free student.
Modern small instruct models score well on verifiable English instructions: Qwen3 4B, 8B and 14B reach 81.2, 83.0 and 84.8 IFEval strict-prompt in non-thinking mode (Qwen3 report). Gemma 3 self-reports higher IFEval under a different scoring setup and a translation score (WMT24++, a chrF-type character F-score) of 46.8 at 4B and 51.6 at 12B (Gemma 3 report); its IFEval should not be compared to Qwen's table. Stacked constraints degrade: on FollowBench, GPT-4's hard satisfaction fell from 84.7% at one constraint to 61.9% at five, and Vicuna-7B from 60.8% to 23.9%; style was GPT-4's easiest category (92.7%), and the authors estimate closed models satisfy about three constraints and open models about two (FollowBench, 2023-era models). One follow-up found plain SFT slightly lowered LLaMA3-8B's FollowBench score while judge-guided DPO raised it (soft-constraint paper, not citation-checked). A style profile of honorifics, puns, register and voice is four or more constraints on top of fidelity, which is why the deterministic layer should take the closed-class flags off the model's plate.
Persona drift is a design constraint, not a blocker. LLaMA2-chat-70B
drifted from its system-prompt persona within eight dialog rounds, with
gpt-3.5-turbo-16k also drifting, and the authors' split-softmax fix
amplifies attention to the system prompt (Li et al., Measuring and
Controlling Persona Drift). A renderer that makes each line an
independent call with the style profile freshly at the end of the
prompt, as Character Card V2's post_history_instructions
does (Character
Card V2 spec), avoids accumulation. Persona vectors steer traits
such as sycophancy and humour in 7-8B open models at some cost to
capability, but were not shown for speaking style (Anthropic,
Persona vectors, not citation-checked).
A third-party llama.cpp benchmark on an RTX 4090 (24 GB; Q4_K_XL, flash attention, model family not named) measured 8B generation at 131 tokens/s and prompt processing at 9,121 tokens/s at 4K context, and 14B at 82.8 and 5,358 (Hardware Corner, secondary, not peer-reviewed). From those figures (my arithmetic, not a measurement), a 600-token prompt and 60-token line costs about 0.5-0.6 s on 8B and 0.9 s on 14B cold, and prefix caching of the shared system prompt cuts prefill further (vLLM prefix caching, listing only). Grammar-constrained JSON decoding costs under 40 microseconds per token mask with XGrammar (XGrammar), which matters for the offline record writer, not the plain-text renderer. UGTLive's author guesses 3-6 s for a whole live OCR-plus-LLM loop (UGTLive), so pre-rendering ahead is what makes any of this feel instant.
Evaluation is where naive setups mislead. Neural QE metrics and LLM judges penalise length: appending one to five error-free segments dropped scores by 4.0-8.0 points for MetricX-24 QE, 1.5-5.6 for CometKiwi, 1.7-5.3 for Gemini AutoMQM and 1.8-8.5 for Gemini as a judge, across about 80% of documents, and MetricX and CometKiwi preferred the shorter of two comparable translations 50.4-60.4% of the time (Penalizing Length). A pirate render is longer and will be marked down for it. Judges prefer their own family, and small judges more so (Llama-3.1-8B bias 21.6% against 0.4% for 70B; Qwen2.5 0.5B 41.7% against 2.1% for 14B) (Beyond the Surface), so the judge must come from neither the teacher's nor the renderer's family. For style accuracy GPT-4 tracked humans best (Pearson 0.92 on English sentiment transfer), but for content preservation BERTScore and BLEURT matched or beat it in several cells (Evaluating TST evaluation), so no single judge covers both. The WebNLG error counts (omitted, wrong, added) are the right shape for checking a styled render against the record's meaning field.
Text licensed CC BY 4.0 (attribution: Hiina, ai.hiina.space). Source: https://github.com/hiinaspace/ai-hiina-space