7 posts tagged with "Corpus pipeline"

A month ago I wrote about adding Canada to the geocoder and seeing it put Toronto in Ohio. In that case, 843,000 correct postcodes never affected the result, because the layer I had built was not the layer that shipped. This post describes a related problem. Last night I added three new countries to the parser. Two of them worked. The third failed a regression check I trusted, and the investigation showed that the addresses were teaching the model something I had not intended.

Feed an address into a sequence labeler and part of it goes missing. The model tags the tokens it recognizes — house number, street, city — and the characters it doesn't understand fall on the floor. decodeAsJSON hands you back a tidy object with every field in its place, and no field in that object tells you the modelshrugged at a fifth of the input.

So you ship that parser. It looks great on the addresses you tested it against. And you have no idea what it's dropping in production, because the output format was built to hide exactly that.

Open the demo and type an address in Kabul, and the demo returns no result. It does not place a pin a few streets off or fall back to the city center, because the gazetteer has no entry for Kabul. Hong Kong, Tirana, Chişinău, and every place in the Democratic Republic of the Congo return no result as well. We had been calling this gazetteer "world coverage" for weeks, and it covered 97 of the world's ~195 countries. The other ninety-odd countries were absent from the file.

We spend effort distrusting numbers that look too good. A validation score that jumps, an accuracy that rounds up suspiciously close to 100 — we've been burned by those, so we poke at them. The number that reports you failed gets a free pass. We assume it's right, because who lies to make themselves look bad?

Our evals did, three times this week. One of them nearly talked us out of a model we should ship. One invented a coverage problem we don't have. And one had us writing "3.3 km" into a model card for a geocoder that puts most addresses within a hundred meters. Each time the fix was the same, and embarrassingly cheap: stop reading the summary row and pull the actual records the summary is averaging over.

Here's a number that should make you nervous: our validation macro-F1 climbed from 0.71 to 0.73 on a retrain we were ready to ship. Instinct reads that as a win: the aggregate went up, so the model must be better, so publish the release.

It wasn't, and it took three probes to prove it. The same model that scored higher on the average had gotten worse at the most basic address there is — a town and a state. This post describes how the average lied to us, and how we caught it.

Our failure taxonomy now ranks boundary instability first among the parser's weaknesses. We spent a night building training data to fix it and then chose not to retrain. Three separate checks each caught a step that looked reasonable and was wrong, and this post is about those checks more than the extract. The checks let us work quickly on the worst weakness without shipping a fix that improves the headline number while introducing hidden errors.

Our training corpus is built from source. The build combines hundreds of millions of address rows from eleven raw data sources and writes them to disk over a long unattended night. Last night's build ran for two and a half hours and then failed on a single row. This post answers three questions. How does one row out of nearly 700 million stop hours of compute? Why, after we fixed the first failure, did the build crash again on the exact same row? What should change once you know that a correct assertion can still be an unsafe one?