---
id: jp-learning-and-anki-tooling/10-open-questions-and-evaluation-plan
title: 'Project status: Japanese immersion learning with Anki and selective glossing:
  10. Open questions and evaluation plan'
report: https://ai.hiina.space/reports/jp-learning-and-anki-tooling.md
section: 10
version: 1
updated: '2026-10-27'
author: Hiina
---

Index of everything on this site: https://ai.hiina.space/index.md

# Project status: Japanese immersion learning with Anki and selective glossing: 10. Open questions and evaluation plan

- Section 10 of 12 of the report "Project status: Japanese immersion learning with Anki and selective glossing" by Hiina (v1, updated 2026-10-27)
- Report overview and contents: https://ai.hiina.space/reports/jp-learning-and-anki-tooling.md
- Previous: https://ai.hiina.space/reports/jp-learning-and-anki-tooling/09-ranking-and-staged-plan.md
- Next: https://ai.hiina.space/reports/jp-learning-and-anki-tooling/11-conclusion.md

## 10. Open questions and evaluation plan

### Open questions

- **Jiten's Anki export.** Does it map onto Lapis fields? How accurate are Jiten's per-title media decks? Its accuracy and data-source pages did not load.
- **Name and sense handling.** How well does an LLM pick JMdict senses and spot character names in anime dialogue? No evaluation exists. The *Katanagatari* set (一寸, character names that are ordinary words, the 9 cards missing sentences) is the first test set.
- **FSRS after Reset.** Does resetting rewritten cards distort the optimiser? Reset keeps history, but the effect on parameter fits is unmeasured.
- **Retention target.** Should immersion-heavy learners run lower desired retention, around 0.85, on show-specific decks? No primary source argues it either way.
- **Linux tooling.** Will GSM's Linux hooking and Wayland capture hold up on the project's Linux desktop, and will the asbplayer Anki add-on work in Firefox on Linux? Both are newer than any guide.
- **Infrastructure.** Will AnkiConnect keep working from its new sr.ht home, or should agent work standardise on ankimcp (Anki must be open either way)?
- **Latency.** What is the per-line latency of a local 7-14B model on a single 24 GB consumer GPU for rung-4 hints? No measurement was found.
- **Kanji.** What to do with the 269 radical notes was not covered by any source; kanji handling is a research gap.
- **Anki time versus immersion.** No source gives a ratio for intermediates. The guides only say to keep Anki subordinate to immersion.

### Evaluation plan

The Bastani result says to measure without the tool in the loop. Each measurement below has that property.

- **Retention:** Anki's true retention by maturity, before and after the changes, plus the `tag:leech` count per optimiser cycle.
- **Sentence cueing:** the word-only probe (the next gate), repeated quarterly.
- **Immersion difficulty:** lookups per 100 tokens and the share of lines needing rung 4 or 5, from overlay logs, tracked per show and over time. A falling lookup rate on comparable shows is the most direct sign of learning.
- **Choosing shows:** Jiten coverage for the next three candidate shows.
- **Comprehension of an unseen episode:** at the start of a new season, watch one episode with Japanese subtitles and no lookups, write a short summary, and compare it with the episode synopsis. Repeat at the same point the next season.
- **Time in Japanese:** hours per week from GSM and asbplayer stats, compared with Anki minutes. If the tooling raises Anki time and lowers immersion time, it is failing on the project's own definition of success.
- **Gloss experiment:** the within-subject randomisation in (d). Words, not sessions, are the unit, which gives a personal test with real statistical power from a single season of watching.

### Research method and coverage

Pages were read through a summarizing fetch tool. Sources that failed to load: animecards.site subpages, learnjapanese.moe anki-settings, the FrequencyMan and FSRS Helper repos, AnkiWeb add-on pages, r/LearnJapanese and r/Anki threads, Jiten's accuracy and data-source pages, jimaku.cc, kitsunekko, and the GitHub API. Community-consensus claims above therefore rest on vnclub, TheMoeWay, Donkuri and tool documentation rather than broad community polling.

---

Text licensed CC BY 4.0 (attribution: Hiina, ai.hiina.space). Source: https://github.com/hiinaspace/ai-hiina-space