16 posts tagged with "Model training"

A month ago I wrote about adding Canada to the geocoder and seeing it put Toronto in Ohio. In that case, 843,000 correct postcodes never affected the result, because the layer I had built was not the layer that shipped. This post describes a related problem. Last night I added three new countries to the parser. Two of them worked. The third failed a regression check I trusted, and the investigation showed that the addresses were teaching the model something I had not intended.

We spend effort distrusting numbers that look too good. A validation score that jumps, an accuracy that rounds up suspiciously close to 100 — we've been burned by those, so we poke at them. The number that reports you failed gets a free pass. We assume it's right, because who lies to make themselves look bad?

Our evals did, three times this week. One of them nearly talked us out of a model we should ship. One invented a coverage problem we don't have. And one had us writing "3.3 km" into a model card for a geocoder that puts most addresses within a hundred meters. Each time the fix was the same, and embarrassingly cheap: stop reading the summary row and pull the actual records the summary is averaging over.

Here's a number that should make you nervous: our validation macro-F1 climbed from 0.71 to 0.73 on a retrain we were ready to ship. Instinct reads that as a win: the aggregate went up, so the model must be better, so publish the release.

It wasn't, and it took three probes to prove it. The same model that scored higher on the average had gotten worse at the most basic address there is — a town and a state. This post describes how the average lied to us, and how we caught it.

Our failure taxonomy now ranks boundary instability first among the parser's weaknesses. We spent a night building training data to fix it and then chose not to retrain. Three separate checks each caught a step that looked reasonable and was wrong, and this post is about those checks more than the extract. The checks let us work quickly on the worst weakness without shipping a fix that improves the headline number while introducing hidden errors.

Two days ago we retired a piece of decode-time implementation we'd been leaning on, and it exposed something the crutch had been hiding: our parser can't read a French address backwards. Last night we tried to teach it to. We got most of the way, hit a wall at 87%, and did the obvious thing and turned the training signal up, fully expecting the last few points to fall out. They didn't. The model got worse, and this post explains how it got worse. The questions on the table: can a pile of reordered examples teach a model where a house number lives? When that pile gets you 80% of the way, can you just add more? And what does the failure tell you about the thing you were training?

Yesterday we wrote about a lookup table that scored a perfect 100 and nearly talked us out of our own architecture. We ended that post with a rule: write the bar down before you look at the score. What we didn't know was that the next 24 hours would test that rule four separate times, and that the fourth test would flip a conclusion we'd been quoting for two weeks.

This post describes shipping v4.2.0: a consolidation, a bar that lowered itself, a consultant who was confidently wrong twice, a capacity wall with a number on it, and a benchmark harness that had been starving our own model for four releases.

This morning we published a post that ended with a tidy rule: some address tags don't want a neural network, they want a lookup table. Country names are a closed list in a known position. Our deterministic matcher scored a perfect 100 on the eval. The retrained model scored a mess. Case closed, we wrote.

By the afternoon we'd reopened the case, and the verdict flipped — hard enough that we've retracted the morning post rather than leave the wrong conclusion lying around for someone to cite. This post describes how a perfect score nearly talked us out of the entire premise of the project.

The last postcode post concluded that the "which country is this" signal has to come from the trained model reading the whole string, because the postcode by itself settles the question less than half the time. That is the expensive version of the feature. In this post we built the country signal into the model. It improved German addresses in native word order by a large margin and did not help the same addresses in a different word order.

We took the postcode's gazetteer membership (the [us, de, fr] answer from last time) and injected it into the model at the postcode token instead of passing it to a regex. The injection is a small additive vector on the hidden state at the position of the five digits, encoding which countries the code could belong to. On German addresses in German word order, it added thirty-five points of locality accuracy and beat Pelias.

The international-order numbers were much worse. With the same model, the same anchor, and the same German cities written house-number-first with the postcode after the city (the way our test feed renders them), the model scored slightly above 50%. On those rows the anchor did slightly worse than no anchor.

The rest of this post addresses three questions:

• When a parser "collapses" on a test, is the parser wrong, or is the test?
• Can you train one model to read an address in any order, or does each order cost you the other?
• What does a learned anchor learn: the meaning you intended, or the position where it appeared in training? This one took three retrains to answer.

We spent a night trying to make our neural address parser less overconfident. The diagnostics showed a different problem: the training data contained no examples of whole categories of address.

This post describes how we optimized the wrong metric and which diagnostics pointed at the right one.

The v0.5.0-a1 tokenizer was trained exclusively on US and French addresses. When it encountered Chinese, Japanese, Korean, Thai, or Arabic text, it fell back to encoding individual bytes — 50-75% of tokens for CJK scripts. Every byte-fallback token is a lost opportunity for the model to learn meaningful subword patterns.

Today we fixed that.