18 posts tagged with "Geocoding"

Every geocoder turns an address into a coordinate. Almost none of them will tell you where that coordinate came from. You get a latitude, a longitude, and a vague confidence enum, and when it's wrong you have no thread to pull — no way to know whether the point came from a federal data release, a county GIS office, or a straight line drawn down the middle of a street. Mailwoman keeps the source on every point. Here's New York, every dot colored by the open dataset it came from.

Three public datasets land on your desk. The national provider registry — NPPES, every NPI in the country. A federal telecom-funding file from the FCC's Rural Health Care program, one part of the Universal Service Fund. A state list of licensed nursing facilities from Texas HHSC. You want to know which records describe the same provider, and not one of the three shares an identifier with the other two. The NPI is internal to NPPES. The funding file keys on its own SPIN. The state list has its own facility ID. There's no crosswalk, because none of the three publishers ever built one.

So you do what everyone does: you start a spreadsheet, you sort by name, and you give up around row 400.

Suppose you have a set of records and no key to join them on. A clinic appears in the federal provider registry, again in a state licensing export, and a third time in a funding-program spreadsheet that someone typed by hand. The files share no identifier. The provider number is internal to one publisher, and the facility ID is internal to another. You must judge which records describe the same place for each pair, repeated a few million times.

The market offers two kinds of tools for this. Each solves part of the problem, and neither solves all of it.

Here are two addresses. Tell me if they're the same place.

123 Main Street, Suite 400, Springfield IL 62704
123 Main St #400, Springfield, Illinois

They are. Now compare these two:

Jyllandsgade 15, 9000 Aalborg
Jyllandsgade 75, 9000 Aalborg

They are different places, 650 meters apart.

A string-similarity matcher scores the first pair, the same place, low because of different punctuation, "Street" vs "St", and a reordered unit. It scores the second pair, different places, at 0.96 because they differ by one character. The usual tool gets both pairs wrong. A threshold change cannot fix this, because string similarity measures the wrong thing.

We turned the demo into a real geocoder. You type an address and get a rooftop coordinate, computed in your browser without a server. The last feature was autocomplete. It completes the city name while you type. We had shipped autocomplete as a command-line tool days earlier and seen it rank San Francisco above San Diego, so we considered it done. We connected the same function to the search box, typed New Yor, and it suggested Denver.

This raised three questions. Why does a function that handles San correctly fail on New Yor? How does completing a word differ from completing the word a person is still typing? How does an autocomplete that knows ten thousand cities fail to complete one of them?

Our resolver scored 93.7% on the metric we'd been quoting for months. On the same addresses, its median answer was 326 kilometers from the truth.

Both numbers are correct. That's the awkward part.

Our neural address parser passes 20.7% of our test suite. The rule-based parser it's meant to replace passes 93.7%. By that scoreboard, we should delete the neural model and go home.

We shipped the neural model instead. Here's why both numbers are true — and why the one that matters points the opposite way.

Who's On First is the best open gazetteer we have, and one of the strangest datasets you'll encounter as a developer. This post covers what makes it hard to use, what makes it useful despite the effort, and the tooling we built inside Mailwoman to work with it.

If you've ever tried to answer "what city is this address in?" programmatically with open data instead of a paid geocoding API, you've probably already run into WOF, and it probably raised some questions.