This is a follow-up to yesterday's post about the v0.5.0 C-train failures. Yesterday we ran four attempts and ruled out three suspects. Today we ran a fifth and ruled out a fourth. One hypothesis remains. The method we used comes directly from software debugging, so this post is written for engineers who haven't run a training campaign before.

If you've ever bisected a software regression, you already understand the method. That includes running git bisect, narrowing a test failure by reverting changes one at a time, or comparing a known-good build with a known-broken build to find which change caused the breakage. The rest is vocabulary.

Who's On First is the best open gazetteer we have, and one of the strangest datasets you'll encounter as a developer. This post covers what makes it hard to use, what makes it useful despite the effort, and the tooling we built inside Mailwoman to work with it.

If you've ever tried to answer "what city is this address in?" programmatically with open data instead of a paid geocoding API, you've probably already run into WOF, and it probably raised some questions.

This is the third post in a series about a training problem we've been chasing. The first two were written for software engineers. This one is for someone who is starting to learn about AI and machine learning. It assumes no jargon and no math beyond high-school algebra, and it uses a problem we ran into this week to show what ML debugging looks like.

If you've been programming for a while but ML feels opaque, this post is for you. Our core technique was to measure which of two training signals the model was following, and that turned out to be much closer to ordinary debugging than the field typically makes it sound.

v0.5.0 was the fresh-slate release, with a new tokenizer, an expanded corpus, a new architecture, and a new reconcile stage. The plan was to bundle several months of structural improvements into one big iteration and pay the integration cost once. Most of it landed as planned, but the classifier did not.

This post walks through the four training attempts the v0.5.0 C-train made overnight, the bisect that ruled out three plausible explanations, and what we think the remaining cause is. It is a companion to the v0.4.0 retrospective. That retrospective describes the same failure shape with a different sequence of diagnostics.

@mailwoman/neural-weights-en-us@v0.4.0 and its fr-fr sibling shipped today as packaged artifacts. The npm publish is a separate manual step. The release has mixed results. It improves fine-grained labels and regresses two coarse labels, and those regressions turned out to be mostly artifacts of how we measured. We set out to combine three orthogonal training improvements into one release, and a divergence pattern we hadn't seen before ruled out almost all of that plan.

This post describes how the campaign went. We're publishing it to explain what the headline numbers mean, and because the way the failures stacked up is relevant if you train your own NER-style models.