Unseen Question Kinds and Coverage Checks / V119.1–V119.4
On 40 questions of kinds never trained, EMMA’s planner with step checks alone committed 30 wrong answers: each step was right, but the plan answered a different question. A check that the plan covers the whole question cut wrong answers to 5, then 4, then 1 of 40 across three fresh sealed sets, with no wrong familiar answer in the final version. V119.1 to V119.3 failed; V119.4 passed.
SCOPE + LIMITATIONS
- The coverage check is lexical, so it has a vocabulary and its misses are unknown words; a learned check is the next step. One family of calculation questions.
- This public edition reviews saved evidence; experiments were not rerun for publication.
CONTROL COMPARISON
Keep the V119 planner unchanged and change only the checks. Score each version on a fresh sealed set of familiar multi-step and direct questions, decline questions, and 40 questions of unseen kinds that should be withheld rather than answered wrongly. Measure false refusals on the 5,238 admitted training plans.
V119.4 passed all five gates. Each earlier version failed on a word the check did not know. Every set was fresh, and earlier sets became development data once scored. Withholding is the safe direction: four multi-step questions with unfamiliar difference wording were withheld in V119.4.
- V119.1 · step checks only
- 30 of 40 unseen-kind answers wrong
- V119.2 · coverage check
- 5 wrong; familiar 0 wrong, 97.6% multi-step
- V119.3 · quantity coverage
- 4 wrong; 1 decline committed
- V119.4 · wording fixes
- 1 wrong; familiar 0 wrong, 90.5% multi-step, 15 of 15 declines
- False refusals on admitted plans
- 25% at the first draft; 1.0% in V119.4
Question and method
Keep the V119 planner unchanged and change only the checks. Score each version on a fresh sealed set of familiar multi-step and direct questions, decline questions, and 40 questions of unseen kinds that should be withheld rather than answered wrongly. Measure false refusals on the 5,238 admitted training plans.
Recorded results
| Condition | Recorded result |
|---|---|
| V119.1 · step checks only | 30 of 40 unseen-kind answers wrong |
| V119.2 · coverage check | 5 wrong; familiar 0 wrong, 97.6% multi-step |
| V119.3 · quantity coverage | 4 wrong; 1 decline committed |
| V119.4 · wording fixes | 1 wrong; familiar 0 wrong, 90.5% multi-step, 15 of 15 declines |
| False refusals on admitted plans | 25% at the first draft; 1.0% in V119.4 |
Comparative standing and locked gates
V119.4 passed all five gates. Each earlier version failed on a word the check did not know. Every set was fresh, and earlier sets became development data once scored. Withholding is the safe direction: four multi-step questions with unfamiliar difference wording were withheld in V119.4.
Interpretation and limitations
The coverage check is lexical, so it has a vocabulary and its misses are unknown words; a learned check is the next step. One family of calculation questions.
SOURCE PROVENANCE
Unseen Question Kinds and Coverage Checks / V119.1–V119.4
LABORATORY REPORT / 2026-10-27SOURCE CHECKSUM / SHA-256
17ab5210f53db370970b17d4e4a26ed6853849d33f55142f431b10e260b93b31Public journal edition reviewed 2026-10-28. Source documents and saved evidence were inspected; experiments were not rerun for this edition. Proprietary implementation code, model binaries, private infrastructure, and detailed machine records are not published here. Journal identifiers are editorial references. Catalog inclusion does not imply qualification or runtime promotion.