The model said P.O. Box all along. We weren't listening.

Last night our release check failed twice before it passed, and both failures had the same cause: the decoder discarded information the model had produced. When a model fails an eval, the usual responses are more data or more parameters. This post is about a third option. It covers three questions: why a model that scored 89 on post-office boxes in validation scored 60 at the check, why fixing that broke French postcodes, and what this shows about where a parser's knowledge lives.

A tag with a 98% failure rate, sorted by punctuation

Our parser had three empty rows on its scorecard: po_box, cedex, and intersections. Last night's release candidate included new training data for all three, and the check had pre-registered floors for each. Intersections scored 100 and cedex scored 96. po_box scored 60.4 against a floor of 70.

We expected the failure to come from a thin layout class or an under-weighted extract. Instead, the failures sorted by a key we had never graded by: punctuation. Plain leaders like PO Box 19 passed at 84%. Dotted leaders (P.O. Box, C.P., B.P.) failed at 98%, and every failure was a truncation at the first period. The model would emit p and stop.

We dumped the raw per-piece labels for P.O. BOX 3274, and the model had labeled every piece correctly: P as po_box at 0.93 confidence, O as po_box at 0.94, and BOX 3274 as po_box at 0.96. The span broke apart at the periods. Our corpus format tokenizes on whitespace and drops standalone punctuation, so it cannot put a label on a dot. None of the 670 million training rows could teach the span to stay whole, and ten times more training data had moved the score by 2.9 points. More repetition cannot fix a gap in the representation.

The decoder merges fragments across punctuation

The fix needed neither new data nor new weights. It is a dozen lines of decode logic that we now call the span bridge. When two fragments have the same tag and only a short run of intra-token punctuation (dots, hyphens, or slashes) separates them, the bridge merges them and absorbs the punctuation the corpus could never label. The model and weights stayed the same, with no GPU time spent. po_box went from 60.4 to 87.

The re-check then failed on French postcodes, and this failure taught us more. The bridge had also been merging across commas. For Sainte-Livrade-sur-Lot 47110, 9016 Route de Lablasie, the model sometimes labeled the house number as a second postcode, and the comma was the only thing keeping the two spans separate. The bridge merged the mistake into 47110, 9016. A period sits inside a surface form, and a comma separates two of them. Once the bridge distinguished the two, French postcodes recovered and po_box rose another two points.

The third battery passed 17 floors out of 17, and the release shipped before sunrise.

Where the knowledge lives

We have learned this more than once. The transformer is one layer of a system, and an eval failure points at the system. The cause may or may not be in the weights. Last month a stubborn tag turned out to be contradictory labels in the corpus. Last night it was a label format that could not represent what the model already knew. In both cases, scaling the model or the data would have cost a week and changed no outcome, because the constraint was in a different layer. A review of forty failing rows exposed it.

The investigation used three full check batteries in one night and retained the pre-registered floor without rounding it up. A check that fails twice and shows why is more useful than several that pass without explanation.

A larger fix is still needed. The punctuation gap is a property of our corpus format, and the proper fix (labels addressed by character offset rather than token index) is a bigger change than one night allows. The bridge is a documented workaround. The scorecard that had three empty rows yesterday has none today, and the parser now outputs "P.O. Box" as one span, matching the per-piece labels the model had produced all along.

Check what your decoder discards.