bench · advisor
The eval set that caught what prompting couldn't hold — frozen OOD curveballs for grounded citation, refusal, and routing
In-distribution receipts hide hint-scaffolding inflation and refusal regressions: the 30B teacher scored 28/28 on the held-out set, then broke its prompt-only contract on the frozen curveball gate (8/21, 3 fabricated private-state rows). This bench packages the held-out set, both sha-pinned OOD curveballs (frozen before the training they gate), and the 182-source corpus manifest — so anyone can reproduce the deltas with a deterministic scorer, no LLM judge.
- Score any lane's citation/refusal/route behavior against the frozen OOD gates with the public deterministic scorer
- Study how pre-registered curveball benches expose prompt-contract failures that held-out sets miss
- Rebuild the Advisor's retrieval packets from the sha-pinned corpus manifest
Audience — Builders of corpus-grounded assistants who want refusal and citation discipline proven on out-of-distribution pretexts, not just held-out accuracy.