18 posts tagged with "Geocoding"

Every geocoder turns an address into a coordinate. Almost none of them will tell you where that coordinate came from. You get a latitude, a longitude, and a vague confidence enum, and when it's wrong you have no thread to pull — no way to know whether the point came from a federal data release, a county GIS office, or a straight line drawn down the middle of a street. Mailwoman keeps the source on every point. Here's New York, every dot colored by the open dataset it came from.

Three public datasets land on your desk. The national provider registry — NPPES, every NPI in the country. A federal telecom-funding file from the FCC's Rural Health Care program, one slice of the Universal Service Fund. A state list of licensed nursing facilities from Texas HHSC. You want to know which records describe the same provider, and not one of the three shares an identifier with the other two. The NPI is internal to NPPES. The funding file keys on its own SPIN. The state list has its own facility ID. There's no crosswalk, because nobody ever built one.

So you do what everyone does: you start a spreadsheet, you sort by name, and you give up around row 400.

You have a pile of records and no key to join them on. A clinic shows up in the federal provider registry, again in a state licensing export, a third time in a funding-program spreadsheet somebody keyed by hand. None of those files share an identifier. The provider number is internal to one publisher, the facility ID to another. So the join you actually want — which of these are the same place — isn't a join at all. It's a judgment call, repeated a few million times.

The market hands you two tools for this, and each one solves a different half of the problem. Neither one finishes.

Here are two addresses. Tell me if they're the same place.

123 Main Street, Suite 400, Springfield IL 62704
123 Main St #400, Springfield, Illinois

Easy — yes. Now these two:

Jyllandsgade 15, 9000 Aalborg
Jyllandsgade 75, 9000 Aalborg

Also easy — no. They're 650 metres apart.

Now imagine a string-similarity matcher looking at those same four lines. The first pair, the same place, scores low: different punctuation, "Street" vs "St", a reordered unit. The second pair, different places, scores 0.96 — one character apart. The tool you'd reach for gets both backwards. This isn't a tuning problem you can threshold your way out of. It's the wrong coordinate system.

We turned the demo into a real geocoder — type an address, get a rooftop coordinate, all in your browser, no server. The last touch was the one that makes a search box feel alive: autocomplete, so the city finishes itself while you type. We already had the autocomplete. We'd shipped it as a command-line tool days earlier, watched it rank San Francisco above San Diego, and called it done. So we dropped the same function into the box, typed New Yor, and it suggested Denver.

The questions that opened up: why does a function that nails San choke on New Yor? What's the difference between completing a word and completing the word a person is in the middle of typing? And how does an autocomplete that knows ten thousand cities fail to finish one of them?

Our resolver scored 93.7% on the metric we'd been quoting for months. On the same addresses, its median answer was 326 kilometers from the truth.

Both numbers are correct. That's the awkward part.

Our neural address parser passes 20.7% of our test suite. The rule-based parser it's meant to replace passes 93.7%. By that scoreboard, we should delete the neural model and go home.

We shipped the neural model instead. Here's why both numbers are true — and why the one that matters says the opposite.

Who's On First is the best open gazetteer we have. It's also one of the strangest datasets you'll encounter as a developer. This post is about what makes it hard to use, what makes it worth the effort, and the tooling we built inside Mailwoman to tame it.

If you've ever tried to answer "what city is this address in?" programmatically, using open data without paying a geocoding API, you've probably already run into WOF. And you probably had some questions.