16 posts tagged with "Evaluation"

Our failure taxonomy now ranks boundary instability first among the parser's weaknesses. We spent a night building training data to fix it and then chose not to retrain. Three separate checks each caught a step that looked reasonable and was wrong, and this post is about those checks more than the extract. The checks let us work quickly on the worst weakness without shipping a fix that improves the headline number while introducing hidden errors.

Two days ago we retired a piece of decode-time implementation we'd been leaning on, and it exposed something the crutch had been hiding: our parser can't read a French address backwards. Last night we tried to teach it to. We got most of the way, hit a wall at 87%, and did the obvious thing and turned the training signal up, fully expecting the last few points to fall out. They didn't. The model got worse, and this post explains how it got worse. The questions on the table: can a pile of reordered examples teach a model where a house number lives? When that pile gets you 80% of the way, can you just add more? And what does the failure tell you about the thing you were training?

Last night our release check failed twice before it passed, and both failures had the same cause: the decoder discarded information the model had produced. When a model fails an eval, the usual responses are more data or more parameters. This post is about a third option. It covers three questions: why a model that scored 89 on post-office boxes in validation scored 60 at the check, why fixing that broke French postcodes, and what this shows about where a parser's knowledge lives.

Yesterday we wrote about a lookup table that scored a perfect 100 and nearly talked us out of our own architecture. We ended that post with a rule: write the bar down before you look at the score. What we didn't know was that the next 24 hours would test that rule four separate times, and that the fourth test would flip a conclusion we'd been quoting for two weeks.

This post describes shipping v4.2.0: a consolidation, a bar that lowered itself, a consultant who was confidently wrong twice, a capacity wall with a number on it, and a benchmark harness that had been starving our own model for four releases.

This morning we published a post that ended with a tidy rule: some address tags don't want a neural network, they want a lookup table. Country names are a closed list in a known position. Our deterministic matcher scored a perfect 100 on the eval. The retrained model scored a mess. Case closed, we wrote.

By the afternoon we'd reopened the case, and the verdict flipped — hard enough that we've retracted the morning post rather than leave the wrong conclusion lying around for someone to cite. This post describes how a perfect score nearly talked us out of the entire premise of the project.

Our resolver scored 93.7% on the metric we'd been quoting for months. On the same addresses, its median answer was 326 kilometers from the truth.

Both numbers are correct. That's the awkward part.