8 posts tagged with "Architecture"

Mailwoman ships two address parsers in the same box.

We call the first one v0. It is our TypeScript port of the Pelias parser — a rules engine: tokenize, classify each token against dictionaries and patterns, solve for the most plausible arrangement under a pile of hand-written constraints. It is fast, deterministic, and the product of years of accumulated postal wisdom. It is also the thing we set out to beat.

The second is the neural classifier — a sequence labeler trained on a BIO-tagged corpus, the one this blog has spent most of its life arguing with.

So: a year in, did the neural net beat the rules parser? The answer is mostly a tie, until the address gets weird — and the weird is where it gets interesting.

Here are two addresses. Tell me if they're the same place.

123 Main Street, Suite 400, Springfield IL 62704
123 Main St #400, Springfield, Illinois

They are. Now compare these two:

Jyllandsgade 15, 9000 Aalborg
Jyllandsgade 75, 9000 Aalborg

They are different places, 650 meters apart.

A string-similarity matcher scores the first pair, the same place, low because of different punctuation, "Street" vs "St", and a reordered unit. It scores the second pair, different places, at 0.96 because they differ by one character. The usual tool gets both pairs wrong. A threshold change cannot fix this, because string similarity measures the wrong thing.

Last night our release check failed twice before it passed, and both failures had the same cause: the decoder discarded information the model had produced. When a model fails an eval, the usual responses are more data or more parameters. This post is about a third option. It covers three questions: why a model that scored 89 on post-office boxes in validation scored 60 at the check, why fixing that broke French postcodes, and what this shows about where a parser's knowledge lives.

Yesterday we wrote about a lookup table that scored a perfect 100 and nearly talked us out of our own architecture. We ended that post with a rule: write the bar down before you look at the score. What we didn't know was that the next 24 hours would test that rule four separate times, and that the fourth test would flip a conclusion we'd been quoting for two weeks.

This post describes shipping v4.2.0: a consolidation, a bar that lowered itself, a consultant who was confidently wrong twice, a capacity wall with a number on it, and a benchmark harness that had been starving our own model for four releases.

This morning we published a post that ended with a tidy rule: some address tags don't want a neural network, they want a lookup table. Country names are a closed list in a known position. Our deterministic matcher scored a perfect 100 on the eval. The retrained model scored a mess. Case closed, we wrote.

By the afternoon we'd reopened the case, and the verdict flipped — hard enough that we've retracted the morning post rather than leave the wrong conclusion lying around for someone to cite. This post describes how a perfect score nearly talked us out of the entire premise of the project.

The last postcode post concluded that the "which country is this" signal has to come from the trained model reading the whole string, because the postcode by itself settles the question less than half the time. That is the expensive version of the feature. In this post we built the country signal into the model. It improved German addresses in native word order by a large margin and did not help the same addresses in a different word order.

We took the postcode's gazetteer membership (the [us, de, fr] answer from last time) and injected it into the model at the postcode token instead of passing it to a regex. The injection is a small additive vector on the hidden state at the position of the five digits, encoding which countries the code could belong to. On German addresses in German word order, it added thirty-five points of locality accuracy and beat Pelias.

The international-order numbers were much worse. With the same model, the same anchor, and the same German cities written house-number-first with the postcode after the city (the way our test feed renders them), the model scored slightly above 50%. On those rows the anchor did slightly worse than no anchor.

The rest of this post addresses three questions:

• When a parser "collapses" on a test, is the parser wrong, or is the test?
• Can you train one model to read an address in any order, or does each order cost you the other?
• What does a learned anchor learn: the meaning you intended, or the position where it appeared in training? This one took three retrains to answer.

For about a month our resolver used one method. It takes a postcode, finds the city polygon that contains the postcode's centroid, and reads off the city. The method got the Netherlands to 95% and Germany to 93%. Then we ran it on Japan, and Japan's gazetteer data has no city polygons.

This post covers two countries where the gazetteer data is sparse, what a geocoder can still do there, and where it stops. We resolved Japan to 94% by replacing the polygon lookup with a postal-authority lookup. Korea had the opposite problem. We could place the coordinate correctly for every postcode, but we could not confirm the name of the place most of the time.

The work raised three questions:

• What do you do when the gazetteer gives you points where you expected shapes?
• Does the method that worked for Japan generalize, or did it work only once?
• What happens when the map is missing data for the part of a country you most need?

We set out to fix a small problem in our address parser and measured a number that told us not to build the larger change we had planned.

The small problem is this. When our postcode extractor sees a five-digit run, it has to decide whether the run is a real postcode or a house number that looks like one. It checks the neighboring words against every country's street vocabulary we know (American, German, and French) at once. This works with three countries. At twenty countries, a German street suffix starts matching an English word by coincidence. We looked for a way to tell the extractor which country's words to check.

That question leads to a larger one, and this post is mostly about the larger one.