16 posts tagged with "Evaluation"

Feed an address into a sequence labeler and part of it goes missing. The model tags the tokens it recognizes — house number, street, city — and the characters it doesn't understand fall on the floor. decodeAsJSON hands you back a tidy object with every field in its place, and no field in that object tells you the modelshrugged at a fifth of the input.

So you ship that parser. It looks great on the addresses you tested it against. And you have no idea what it's dropping in production, because the output format was built to hide exactly that.

Type Vienna, Austria into our geocoder and, until this week, it answered with a confident set of coordinates: 39.32, −81.54. That's Vienna, West Virginia — population about ten thousand, a few miles up the Ohio River from Parkersburg. The capital of Austria is 7,500 kilometers and one ocean away, and the geocoder had no doubt whatsoever.

Last night the plan was simple: mailwoman was missing a pile of non-US coverage, and we were going to fix the parser to close the gap. By morning we'd shipped something better than planned, killed the original idea, and caught ourselves about to spend a GPU budget on a bug that lived in a symlink. Here's how a night of "fix the model" turned into "stop trusting your evals."

You've got a hundred thousand addresses to reconcile. Two databases, the same clinics and providers scattered across both, each one spelled a dozen ways: abbreviated here, reordered there, a postcode dropped, a suite number glued to the street. You run them through a geocoder, match on the resolved coordinate, and it works. Mostly. Some fraction land on the wrong building, the wrong block, the wrong town, and the geocoder won't tell you which fraction. It hands back a pin for every row and the same silent confidence for all of them: none.

That's the gap we set out to close. Every geocoder chases accuracy, be-right-more-frequently, and so do we. The piece that few geocoders hand you is the one underneath it: a number on each answer that tells you which ones to trust, so you can keep the good ones and send the rest to a human.

Our geocoder had beaten Nominatim across the United States by fifteen points, so we ran the same benchmark on Europe and expected a similar result. Europe produced a double-digit loss.

A loss to Nominatim would have been acceptable, because Nominatim covers the whole planet on community-contributed data and is the standard to beat. We could not explain the loss, though. We knew our parser was not ten points worse in Europe than in America, so we investigated what the gap consisted of.

This post covers the cause, the fixes, and the problems the fixes did not solve. For the European comparison we added a third system: Pelias, through geocode.earth. geocode.earth hosts Pelias on Elasticsearch. Pelias turned out to be the stronger competitor, and this post states where it still beats us.

Our parser scores 28% on Spanish addresses. By the metric that reads its labels, it is, on paper, broken in a country of forty-seven million people.

That metric is measuring the wrong thing. The moment we measured the right one, the panic fell apart.

Overnight I taught the geocoder Canada. Or I thought I did. I pulled 843,000 Canadian postcodes, computed a centroid for every one, spot-checked the result — M5H 2N2, downtown Toronto, 43.652, −79.382, dead on — and validated the database every way I could read it: every postcode present, every coordinate right, and no other data disturbed. Green, top to bottom. This morning, before flipping it live, I asked the demo to find a Toronto address. It dropped the pin in Ohio.

So, the questions for the morning. How does a database that passes every check still land the answer 600 kilometers wrong? What does it mean to "validate" a thing? And why is Toronto in Ohio?

We spend effort distrusting numbers that look too good. A validation score that jumps, an accuracy that rounds up suspiciously close to 100 — we've been burned by those, so we poke at them. The number that reports you failed gets a free pass. We assume it's right, because who lies to make themselves look bad?

Our evals did, three times this week. One of them nearly talked us out of a model we should ship. One invented a coverage problem we don't have. And one had us writing "3.3 km" into a model card for a geocoder that puts most addresses within a hundred meters. Each time the fix was the same, and embarrassingly cheap: stop reading the summary row and pull the actual records the summary is averaging over.

Here's a number that should make you nervous: our validation macro-F1 climbed from 0.71 to 0.73 on a retrain we were ready to ship. Instinct reads that as a win: the aggregate went up, so the model must be better, so publish the release.

It wasn't, and it took three probes to prove it. The same model that scored higher on the average had gotten worse at the most basic address there is — a town and a state. This post describes how the average lied to us, and how we caught it.

Mailwoman ships two address parsers in the same box.

We call the first one v0. It is our TypeScript port of the Pelias parser — a rules engine: tokenize, classify each token against dictionaries and patterns, solve for the most plausible arrangement under a pile of hand-written constraints. It is fast, deterministic, and the product of years of accumulated postal wisdom. It is also the thing we set out to beat.

The second is the neural classifier — a sequence labeler trained on a BIO-tagged corpus, the one this blog has spent most of its life arguing with.

So: a year in, did the neural net beat the rules parser? The answer is mostly a tie, until the address gets weird — and the weird is where it gets interesting.