Markdown version: https://ai.hiina.space/reports/jp-learning-and-anki-tooling/10-open-questions-and-evaluation-plan.md · index
Index of everything on this site: https://ai.hiina.space/index.md
Project
status: Japanese immersion learning with Anki and selective glossing:
10. Open questions and evaluation plan
10. Open questions and
evaluation plan
Open questions
- Jiten's Anki export. Does it map onto Lapis fields?
How accurate are Jiten's per-title media decks? Its accuracy and
data-source pages did not load.
- Name and sense handling. How well does an LLM pick
JMdict senses and spot character names in anime dialogue? No evaluation
exists. The Katanagatari set (一寸, character names that are
ordinary words, the 9 cards missing sentences) is the first test
set.
- FSRS after Reset. Does resetting rewritten cards
distort the optimiser? Reset keeps history, but the effect on parameter
fits is unmeasured.
- Retention target. Should immersion-heavy learners
run lower desired retention, around 0.85, on show-specific decks? No
primary source argues it either way.
- Linux tooling. Will GSM's Linux hooking and Wayland
capture hold up on the project's Linux desktop, and will the asbplayer
Anki add-on work in Firefox on Linux? Both are newer than any
guide.
- Infrastructure. Will AnkiConnect keep working from
its new sr.ht home, or should agent work standardise on ankimcp (Anki
must be open either way)?
- Latency. What is the per-line latency of a local
7-14B model on a single 24 GB consumer GPU for rung-4 hints? No
measurement was found.
- Kanji. What to do with the 269 radical notes was
not covered by any source; kanji handling is a research gap.
- Anki time versus immersion. No source gives a ratio
for intermediates. The guides only say to keep Anki subordinate to
immersion.
Evaluation plan
The Bastani result says to measure without the tool in the loop. Each
measurement below has that property.
- Retention: Anki's true retention by maturity,
before and after the changes, plus the
tag:leech count per
optimiser cycle.
- Sentence cueing: the word-only probe (the next
gate), repeated quarterly.
- Immersion difficulty: lookups per 100 tokens and
the share of lines needing rung 4 or 5, from overlay logs, tracked per
show and over time. A falling lookup rate on comparable shows is the
most direct sign of learning.
- Choosing shows: Jiten coverage for the next three
candidate shows.
- Comprehension of an unseen episode: at the start of
a new season, watch one episode with Japanese subtitles and no lookups,
write a short summary, and compare it with the episode synopsis. Repeat
at the same point the next season.
- Time in Japanese: hours per week from GSM and
asbplayer stats, compared with Anki minutes. If the tooling raises Anki
time and lowers immersion time, it is failing on the project's own
definition of success.
- Gloss experiment: the within-subject randomisation
in (d). Words, not sessions, are the unit, which gives a personal test
with real statistical power from a single season of watching.
Research method and coverage
Pages were read through a summarizing fetch tool. Sources that failed
to load: animecards.site subpages, learnjapanese.moe anki-settings, the
FrequencyMan and FSRS Helper repos, AnkiWeb add-on pages,
r/LearnJapanese and r/Anki threads, Jiten's accuracy and data-source
pages, jimaku.cc, kitsunekko, and the GitHub API. Community-consensus
claims above therefore rest on vnclub, TheMoeWay, Donkuri and tool
documentation rather than broad community polling.
Text licensed CC BY 4.0 (attribution: Hiina, ai.hiina.space). Source:
https://github.com/hiinaspace/ai-hiina-space