666,000 addresses and not one country

A month ago I wrote about adding Canada to the geocoder and seeing it put Toronto in Ohio. In that case, 843,000 correct postcodes never affected the result, because the layer I had built was not the layer that shipped. This post describes a related problem. Last night I added three new countries to the parser. Two of them worked. The third failed a regression check I trusted, and the investigation showed that the addresses were teaching the model something I had not intended.

The easy part

The parser reads a raw string such as 2835 Woodhaven Rd, Shirley, V9Z1G6 and labels every piece: house number, street, city, and postcode. It is a neural sequence model, and it learns a new country from a few hundred thousand real examples. We had a new source for those examples: Overture's open addresses theme, with millions of rows per country, a permissive license, and no API key. Canada returned sixteen million addresses, Mexico thirty million, and Brazil ninety million.

Canada and Mexico trained without problems. Mexico showed the main improvement: the model's accuracy on Mexican cities went from mediocre to good. It now reads Calle Zeverino León 749, 28809 Manzanillo and labels Manzanillo correctly instead of guessing. The model already handled Canada reasonably well, because Canadian addresses resemble American ones, so adding Canada neither cost nor gained anything measurable. Both countries went into one training run, and the whole battery passed. I then prepared Brazil as the third country.

The guard that said no

Every model change runs a check before it can reach production. Part of the check confirms that the new data worked. The required part checks whether anything else got worse. We keep a frozen set of graded addresses and require that no category regress by more than two points. The check is simple, and it is what lets us move quickly without shipping unnoticed regressions.

Brazil built and trained without errors. The check then reported one failure. Country recall, the model's ability to extract the country from an address that includes one, had dropped more than five points. Every other category (cities, streets, postcodes, and house numbers) stayed flat. Only country dropped, and by enough to fail.

The earlier run explained the pattern. The run with only Canada and Mexico had lowered country recall by about a point and a half. That drop was under the threshold and easy to dismiss as noise. Brazil enlarged the existing problem. The two runs showed a lower country-label rate after more addresses were added.

What the addresses forgot to say

I then inspected the training rows themselves instead of the model. None of the 666,000 new Canadian, Mexican, and Brazilian rows contained a country.

That is normal for the source. An Overture address is street, city, postcode. The country is implied by the file it came from, in the same way that a domestic letter omits the country from the envelope. Each row is correct on its own. A model, however, learns from the whole set. I had given it two-thirds of a million addresses, and none of them included a country. Trained on that data at the weight I had given it, the model learned that addresses do not include their country, in addition to learning three new countries. When the graded set gives it 1600 Pennsylvania Ave NW, Washington, DC, USA, which does include a country, the model has learned to skip the USA token.

I had checked that the new rows had the tags I wanted: street, city, and postcode, all present and correct. I never checked for the tag that was missing, and the missing tag caused the regression.

The fix is old, and we already own it

We built the fix for this a year ago and did not apply it to the new pipeline. Our older locale recipe, the one behind the European countries, renders addresses in their native form and also renders a share of them in an international form, the kind written on a parcel crossing a border: 12 Rue de Rivoli, Paris, France. That form includes the country deliberately, and it is why adding France never taught the model to drop France. The new Overture pipeline skipped that step and used the addresses exactly as given, without countries.

Brazil therefore needs only a small change, and the design stays as it is. The change is to render a fraction of the new rows with the country included, so the training set stops teaching the model to omit it. Canada and Mexico ship as they are, because that run passed. Brazil waits for the next build, and the fix is smaller than it seemed at 4 a.m.

The lesson, in a sentence

June's lesson was to grade the coordinate the user sees rather than the layer you happened to build. This one is related: a model also learns from what your training data consistently omits. Every field you consistently leave out is a field you are teaching the model to leave out. Coverage includes what you put in the corpus and also what you leave out. The model can learn an omission as a rule when the omission repeats across a million rows.

A simple two-point regression check caught this before any user saw it. The same check has now caught two different problems of this kind.