Four numbers tried to lie to us in 24 hours. We shipped anyway.

Yesterday we wrote about a lookup table that scored a perfect 100 and nearly talked us out of our own architecture. We ended that post with a rule: write the bar down before you look at the score. What we didn't know was that the next 24 hours would test that rule four separate times, and that the fourth test would flip a conclusion we'd been quoting for two weeks.

This post describes shipping v4.2.0: a consolidation, a bar that lowered itself, a consultant who was confidently wrong twice, a capacity wall with a number on it, and a benchmark harness that had been starving our own model for four releases.

The trade

We consolidated every proven change of the parity campaign into one training run: the unit extract, the affix extract, the country gazetteer anchor, the multi-locale balance. The core tags soared. Locality up thirteen points, region up eleven, country from no score to 87. And the street-affix split crashed to 27. A solo run had taught that split to 78.

If you've trained multi-task models, you know this trade. Capacity is a blanket that's slightly too small. Pull it up to your chin and your feet get cold.

Lie number one: the bar that lowered itself

While writing up the check criteria for the fix attempts, we restated the targets from memory. Affix ≥78 became "≥72". Unit 92 became 91. And the US street floor, the one number that would have exposed a regression hiding in every single consolidation run, fell out of the table entirely.

No one had chosen to change the targets. The drift started when we restated numbers from memory. Each retelling moved the targets toward the system we had already built. The operator caught it by asking one question about one number. The fix cost ten minutes, and the rule it produced is now non-negotiable: the training config is the canonical check, and any change to a bar is a stated decision with a reason, in writing, before the eval runs. When a model misses the bar, you confront the miss: re-baseline it with reasons, or iterate. Re-describing it as a pass is how you wake up six weeks later wondering when street tagging got worse.

Lies two and three: the consultant's confidence

We consult an external model on consequential forks, and it warrants its keep frequently enough that we keep asking. This time it was sure, twice, with numbers. The affix crash was a scheduling problem rather than a capacity problem. Five-times density would clear 72. And the 75 we'd seen in a quick diagnostic? Not a transient.

Both predictions died by experiment, and we know that precisely because we spent five minutes of GPU on a cheap diagnostic before each expensive run instead of trusting either of us. Five-times density plateaued at 65. The 75 was a transient: hold the density long enough to reach it and the peak decays under your feet, 75 down to 53, while the pressure collapses French region tagging from 25 to 5. The blanket is too small, and yanking it hard enough to cover your feet tears it somewhere you weren't looking.

So that's a capacity wall with a number on it: 29 million parameters cannot hold the affix split and the rest of the distribution at once. It can visit. It can't stay. We wrote months ago that the first model-size increase would have to warrant its way in with evidence, and this is what earning it looks like. The escalation now sits in an issue with a pre-registered check and a cheap probe to try first — a dedicated affix head — before anyone reaches for more parameters.

The miss we shipped

So we shipped the trade. v4.2.0 is the consolidation's strongest stable variant, and its ledger entry records exactly what it gave up: street down 2.3, unit down 1.7, the affix split present but at 65-level, fluent nowhere near its solo 78. The same entry records what everyone got: locality +13, region +11, country from absent to 89.8, French house numbers at their best ever, German locality beating the system we're chasing.

Name your regressions in the model card and something nice happens downstream: the next regression has a recorded baseline to measure against. The cards that only list wins go stale the moment anything moves.

Lie number four: the harness was starving our model

Hours after shipping, one number kept itching. Our capability arenas, the unbiased head-to-head against the rules parser, showed the new model slightly down on whole-parse accuracy. Plausible, right? We'dtraded street precision away, and we very nearly filed it under the stated costs.

Then we checked how the arena harness loads models. Well. It had no way to feed the gazetteer lexicon or the postcode-anchor channel. Every anchor-trained model we've benchmarked in those arenas — across four releases — was graded with those input channels zeroed out. We had been publishing numbers for a parser with its eyes taped shut and treating them as the parser.

Fed its actual shipping configuration, v4.2.0 doesn't trail the rules system on clean, canonical addresses. It beats it: 41% to 29%, with the noisy-input lead stretching to 32 points. "Rules win on clean input" had been our routing principle, our documented truth, the premise of a planned arbitration layer's priors. It was a measurement artifact. The model had been better than its own scoreboard since before we knew to ask.

Calibrate the instrument too

Four numbers lied to us in 24 hours: a check that softened itself in retelling, a consultant's confident extrapolation, a transient peak dressed as an equilibrium, and a benchmark grading a handicapped model. None of them lied maliciously. Every one of them was plausible, and plausible is the only kind of wrong number that survives long enough to cost you something.

The implementation that caught them is boring on purpose. Bars written down before looking. Cheap diagnostics before expensive commitments. Misses confronted in writing. And a suspicious habit that other teams can adopt: whenever a result confirms what you already believe, or condemns what you built, go read how the measurement was taken before you react to it.

The lookup table from yesterday's post scored 100 and was wrong. The arena scored our model low and was wrong. The instrument is part of the experiment. Calibrate accordingly.