843,000 postcodes and no Canada
Overnight I taught the geocoder Canada. Or I thought I did. I pulled 843,000 Canadian postcodes, computed a centroid for every one, spot-checked the result — M5H 2N2, downtown Toronto, 43.652, −79.382, dead on — and validated the database every way I could read it: every postcode present, every coordinate right, and no other data disturbed. Green, top to bottom. This morning, before flipping it live, I asked the demo to find a Toronto address. It dropped the pin in Ohio.
So, the questions for the morning. How does a database that passes every check still land the answer 600 kilometers wrong? What does it mean to "validate" a thing? And why is Toronto in Ohio?
The build that felt right
The task was simple enough: add Canada to the map. The obvious change is postcodes — they're precise, they're public, and Overture ships them. I'd done exactly this for a dozen European countries the night before, same tool, same shape: read the country's address points, group by postcode, take a trimmed centroid, write one row per code. Canada came back with 843,000 of them, alphanumeric A1A 1A1 codes that survive the build intact, each pinned to its own block. I spot-checked Toronto's financial district and it sat exactly where it should. I diffed the new gazetteer against the old — US postcodes untouched, every European set intact, 843,000 new Canadian rows. Every number I could check, I checked, and every one was right.
That is the trap. I had checked everything I'd built.
The check that grades the pin
There's one test in the stack that doesn't care what I built. It loads the actual demo in a real browser, types an address into the box, and reads the coordinate off the map. The pin the user would see. The whole point of it is to grade the assembled answer rather than any single piece of the implementation. Before promoting the new gazetteer I added a Canadian case to it: 100 Queen Street West, Toronto, ON M5H 2N2, and assert the pin lands somewhere in Toronto.
Four cases passed — Chicago, a Berlin postcode, the White House, all green. The fifth came back 40.458, −80.6. I had to look it up. That's Toronto, Ohio: population five thousand, on the bank of the Ohio River, about as far from the address I typed as Madrid is from London.
Why the perfect postcode never got a vote
The gazetteer has two layers. There's the postcode layer I'd rebuilt, and underneath it the admin layer — the one that records a place's country and population. So I went and counted the admin layer: 1.7 million places, twenty-six countries. The number of Canadian ones is zero. Toronto, Montréal and Vancouver are all missing — Canada isn't in it at all. We'd built the gazetteer out of the European set, the US, and the CJK countries, and no build step had ever folded Canada in.
Now watch what that does. The parser reads "Toronto" off the address and hands it to the resolver. The only Toronto the resolver has ever heard of is the one in Ohio, so that's the city it picks. And then the cleverest piece of the cascade turns around and finishes the job. We added that piece a couple of weeks ago to keep Berlin's 10115 — which is also a perfectly good New York ZIP — from dragging German addresses to Manhattan: resolve the city first, then let that city restrict the postcode to its own country. It's a good rule. Here it's a guillotine. The city resolved to the United States, so the correct Canadian postcode is now "from the wrong country," and out it goes. All 843,000 of my perfect coordinates never get a vote, because the city votes first and the city is in Ohio.
I validated the layer I made rather than the one that ships
The lesson is an old one. I validated the postcodes. The postcodes were never the problem. I checked the thing I'd built and not the thing the user receives — the coordinate, the pin, the assembled output of every layer working together. Each green check was accurate and beside the point, because not one of them graded the only number that matters.
This keeps happening to us, in different forms. A model's per-tag F1 climbs while the coordinate it produces goes nowhere. A reconcile pass improves a label and breaks the geocode that depended on it. The discipline that survives every version of it is the same: grade the assembled coordinate against the truth rather than the component you happened to touch. The e2e check isn't ceremony. It's the one thing in the stack that reads the pin the user sees, and it's the only thing that caught this.
What it costs, and what it doesn't
The accounting: the night wasn't wasted. The 843,000 centroids are correct and sitting ready, and the tooling that made them is the same tooling that'll make the next country. But the premise was half a country short. A country, it turns out, is its admin skeleton first — the cities and regions that anchor a name to a place — and a postcode only means something once that skeleton is there to hang it on. Fold Canada's divisions into the gazetteer the way we folded Europe's, and the postcodes I built light up at no added cost. Ship them on their own and all you've done is hand the resolver a stack of right answers to a question it can't ask.
So we didn't promote it. The map still doesn't have Canada — but it knows it doesn't now, and so do we. Build the test that grades the pin your user sees, and run it before you trust the green checkmarks upstream of it. They'll tell you the data is right. The pin tells you whether you are.