Here are two addresses. Tell me if they're the same place.

123 Main Street, Suite 400, Springfield IL 62704
123 Main St #400, Springfield, Illinois

They are. Now compare these two:

Jyllandsgade 15, 9000 Aalborg
Jyllandsgade 75, 9000 Aalborg

They are different places, 650 meters apart.

A string-similarity matcher scores the first pair, the same place, low because of different punctuation, "Street" vs "St", and a reordered unit. It scores the second pair, different places, at 0.96 because they differ by one character. The usual tool gets both pairs wrong. A threshold change cannot fix this, because string similarity measures the wrong thing.

We turned the demo into a real geocoder. You type an address and get a rooftop coordinate, computed in your browser without a server. The last feature was autocomplete. It completes the city name while you type. We had shipped autocomplete as a command-line tool days earlier and seen it rank San Francisco above San Diego, so we considered it done. We connected the same function to the search box, typed New Yor, and it suggested Denver.

This raised three questions. Why does a function that handles San correctly fail on New Yor? How does completing a word differ from completing the word a person is still typing? How does an autocomplete that knows ten thousand cities fail to complete one of them?

Two days ago we retired a piece of decode-time implementation we'd been leaning on, and it exposed something the crutch had been hiding: our parser can't read a French address backwards. Last night we tried to teach it to. We got most of the way, hit a wall at 87%, and did the obvious thing and turned the training signal up, fully expecting the last few points to fall out. They didn't. The model got worse, and this post explains how it got worse. The questions on the table: can a pile of reordered examples teach a model where a house number lives? When that pile gets you 80% of the way, can you just add more? And what does the failure tell you about the thing you were training?

Our training corpus is built from source. The build combines hundreds of millions of address rows from eleven raw data sources and writes them to disk over a long unattended night. Last night's build ran for two and a half hours and then failed on a single row. This post answers three questions. How does one row out of nearly 700 million stop hours of compute? Why, after we fixed the first failure, did the build crash again on the exact same row? What should change once you know that a correct assertion can still be an unsafe one?

Last night our release check failed twice before it passed, and both failures had the same cause: the decoder discarded information the model had produced. When a model fails an eval, the usual responses are more data or more parameters. This post is about a third option. It covers three questions: why a model that scored 89 on post-office boxes in validation scored 60 at the check, why fixing that broke French postcodes, and what this shows about where a parser's knowledge lives.

Yesterday we wrote about a lookup table that scored a perfect 100 and nearly talked us out of our own architecture. We ended that post with a rule: write the bar down before you look at the score. What we didn't know was that the next 24 hours would test that rule four separate times, and that the fourth test would flip a conclusion we'd been quoting for two weeks.

This post describes shipping v4.2.0: a consolidation, a bar that lowered itself, a consultant who was confidently wrong twice, a capacity wall with a number on it, and a benchmark harness that had been starving our own model for four releases.

This morning we published a post that ended with a tidy rule: some address tags don't want a neural network, they want a lookup table. Country names are a closed list in a known position. Our deterministic matcher scored a perfect 100 on the eval. The retrained model scored a mess. Case closed, we wrote.

By the afternoon we'd reopened the case, and the verdict flipped — hard enough that we've retracted the morning post rather than leave the wrong conclusion lying around for someone to cite. This post describes how a perfect score nearly talked us out of the entire premise of the project.

Our resolver scored 93.7% on the metric we'd been quoting for months. On the same addresses, its median answer was 326 kilometers from the truth.

Both numbers are correct. That's the awkward part.

There is a particular kind of engineering misery where you fix a bug three times and it never gets better, because the bug is in your ruler. This is that story.

Our neural parser handles German two ways. Native order — Hauptstraße 5, 10115 Berlin — is the layout real German feeds and real German people use. International order — 5 Hauptstraße, Berlin, 10115 — is the Americanized layout our evaluation set happens to ship. For months, international-order German "collapsed": locality accuracy sat around 44% while native cleared 80%. We had a story for it. The postcode anchor — a side-channel that feeds the model a country hint derived from the postcode — sits at the trailing postcode. In international order that postcode lands on the far side of the locality from where it's needed. Plausible. So we retrained.

Ask a geocoder for "Berlin" and it has to make a choice. There's the one in Germany, There's also Berlin, New Hampshire (population nine thousand and change), Berlin, Wisconsin, Berlin, Connecticut, and a dozen more scattered across the United States like the name was on sale. The parser hands you the word Berlin tagged as a locality. Something downstream has to decide which dot on the map that is. How would you even know if it picked right?

For a long time our answer was a scorecard that checked the name. Did the resolved place's name equal the expected name? Tick. Move on. It is a completely reasonable thing to measure, and it was lying to us for months.