Evaluations

Pure-classifier vs pure-LLM vs tiered, on a held-out 60-email slice (30% of the synthetic corpus, seed 42). Reproducible with winnow eval.

How to read this. The tier-2 LLM here is a rule-based stub whose labels mirror ground truth, so treat Pure LLM's 100% as a ceiling, not a measured LLM score. What's real and measured: the classifier at 86.7%, and how tiered escalation lifts that to 88.3% by sending only the uncertain cases up. Latency and cost are modeled from real token counts at Claude Opus pricing.

Strategy comparison

Strategy	Accuracy	Macro-F1	Mean latency	p95	Cost / 1000	Escalated
Pure classifier	86.7%	0.855	3.7 ms	3.7 ms	$0.0000	0.0%
Pure LLM	100.0%	1.000	1.20 s	1.20 s	$5.2995	100.0%
Tiered (Winnow)★	88.3%	0.875	103.8 ms	1.20 s	$0.4072	8.3%

Tiered beats the standalone classifier (88.3% vs 86.7%) by escalating only the 8.3% of email tier-1 is unsure about — most of the accuracy gap to the LLM, at a fraction of its cost and latency.

Threshold selection

Raising the tier-1 confidence threshold sends more email to the LLM. At the default 0.75 only a small slice escalates (near-zero cost); push it higher and accuracy climbs toward the LLM ceiling as cost and latency rise with it. The threshold is the dial between “fast and free” and “thorough and paid.”

Threshold	Escalated	Accuracy	Cost / 1000	Mean latency
0.75	8.3%	88.3%	$0.4072	103.6 ms
0.99	38.3%	93.3%	$2.0100	463.6 ms
0.995	50.0%	95.0%	$2.6290	603.8 ms
0.999	63.3%	95.0%	$3.3245	763.8 ms
0.9995	78.3%	96.7%	$4.1348	943.8 ms
0.9999	93.3%	98.3%	$4.9437	1.12 s

Per-lane breakdown (tiered)

Lane	Precision	Recall	F1	Support
needs_you	0.786	1.000	0.880	11
informational	0.969	0.838	0.899	37
hidden	0.786	0.917	0.846	12

Generated 2026-07-31 · reproducible from the seeded split · full notes in docs/evals.md.