Index of everything on this site: https://ai.hiina.space/index.md
The benchmarks exist, but they all test reading of human-made comics. On MangaVQA, Gemini 2.5 Flash scored 7.26/10, GPT-4o 6.00 and Claude Sonnet 4.5 5.84. On the strict MangaOCR detection-plus-recognition metric, all three scored 0.0 Hmean, against 71.5 for a fine-tuned Qwen2.5-VL-7B (MangaVQA). On ComicVQA, the best proprietary model reached 62.6% on missing-panel prediction and 46.4% on panel sorting, against humans above 83%. The authors found models lean on coarse temporal cues (ComicVQA). CHROMIC reports that even the strongest models struggle with chronological reasoning and invent fine visual detail (CHROMIC). On AI4VA-FG, Gemini-2.5-Pro led six of seven tasks, yet most models sat near chance on depth comparison, panel reordering and character counting (Zooming into Comics). PixelHumor found models default to left-to-right, top-to-bottom order and miscount panels (PixelHumor). Splitting panels into separate images hurt every model on YesBut (YesBut). Speaker attribution is fragile too: on CoMix, a nearest-character heuristic beat Magi on three of four subsets (CoMix). Every one of these numbers predates the current frontier. I found no published comic-benchmark results for Claude 5-era, GPT-6-era or Gemini 3 models. Treat them as lower bounds, and run a control (below) rather than assuming.
Zoom helps, but the headline numbers are an upper bound. The large gains in "Zooming into Comics" came from manual, ground-truth crops to the relevant panels. GPT-4o's action recognition rose from 31.97 to 55.10, Gemini-2.5-Flash's depth comparison from 56.31 to 78.64, and Qwen2.5-VL-7B's panel understanding from 41.33 to 58.67. The authors' trained zoom model, which predicts its own boxes, gained only 10 points on panel understanding (to 51.33). It did not transfer to MangaVQA and did not fix counting (Zooming into Comics). This shapes the design. The reader does not need to find the right zoom, because the pipeline owns the page geometry. The compositor knows every panel polygon and balloon box, so it can hand the reader exact crops. That puts the pipeline near the oracle end of the gap for free. The pipeline also knows every word of lettering, because it rendered it. So check legibility deterministically, with OCR per balloon crop diffed against the script, and keep the VLM's job to narrative and staging, where the OCR weakness matters less.
Judge validity is moderate where it has been measured, and unmeasured where this design needs it. CANVAS validated storyboard consistency judges against human ratings. The independent judges reached Pearson 0.66-0.71 (Gemini-2.5-Flash 0.67-0.71, GPT-5 0.66-0.70). The low end, 0.61-0.65, came from Gemini-3-Pro-Image, which is also CANVAS's generator backbone. That is a same-family judge doing worse, and it matches the motivating observation that a model from a different family critiques better than the author's own (CANVAS). ViStoryBench's automatic character-similarity metric reached Pearson 0.80 with human ratings, style 0.60, and aesthetics only 0.54 (Kendall 0.26). The benchmark explicitly does not support multi-panel comic layouts (ViStoryBench). PPTAgent's GPT-4o slide judge hit 0.71 Pearson with kappa 0.59 (PPTAgent). Crafter's VLM judge for scientific figures agreed with the human majority on 72% of pairs (Crafter listing). A manga translation-judging paper found that adding the page image raised error detection from 57.9% to about 81%. Its six human verifiers agreed at only Fleiss' kappa 0.266 (arXiv 2610.17908). The general literature supports separating the two roles. LLM evaluators recognize and favour their own outputs (Panickssery et al.), and models struggle to self-correct without an external signal (Huang et al.). Both findings are on text, so cross-family VLM critique of images is a sensible precaution, not a measured result.
Comics theory works as a rubric, as long as its labels are treated as soft. McCloud's six transition types have been used to annotate 2,228 Manga109 panel pairs. Action-to-action made up 33.2%, subject-to-subject 20.4%, non-sequitur 15.1%, moment-to-moment 12.6%, scene-to-scene 10.1% and aspect-to-aspect 8.3%. Three student annotators agreed at Cohen's kappa only 0.524-0.774 (arXiv 2312.08720). That kappa is roughly the ceiling for any VLM-vs-human agreement on transition labels. So use transitions in two ways. The storyboarder declares an intended transition per gutter. The reader infers one, and a mismatch flags the gutter for attention without proving it wrong. The transition distribution also works as a stylistic sanity check against manga norms. Cohn's Visual Language Research Corpus offers open annotations of about 45K panels, including character, space and time changes between panels, which makes a more empirical scheme than McCloud's (VLRC). One generator has encoded Cohn's arc (Establisher, Initial, Peak, Release) alongside McCloud transitions, with qualitative results only (Chen & Jhala). I found no paper that puts McCloud or Cohn into an LLM or VLM prompt as a rubric. In practice, give the author and storyboarder a distilled summary of Understanding Comics in the project's own words (closure across the gutter, transition choice, panel time, word-picture balance), plus the Manga109 transition statistics. Give the reader only the rubric, never the script.
Text licensed CC BY 4.0 (attribution: Hiina, ai.hiina.space). Source: https://github.com/hiinaspace/ai-hiina-space