Benchmarks

Each number on these pages comes from a script published next to it. The inputs and the result file are published there too, so you can re-run the code yourself.

Two panels are published so far. Each page presents a small panel and states that its result is not a national accuracy figure.

I want to see the accuracy a national register produces

France has a national registerinput modeThe Decision-A register switch: 'fragmented' (human-typed fragments — feeds the evidence channels) vs 'formatted' (complete records — runs the trained absence identity). Explicit on CLI/API; per-endpoint defaults (batch→formatted, autocomplete→fragmented); kind-derived otherwise., the Base Adresse Nationale (BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure.), and the French rooftoprooftopGeocoding precision at the building or parcel level — coordinates within a few meters — the highest tier of the geocode cascade. Sourced from address-point and situs data. tier is built from it. The panel renders a hundred BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure. addresses in two surface forms and grades them against BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure.'s own coordinates. In the canonical order, 96 of 100 land within a kilometer. With the postcodepostcodeThe country-specific postcode (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. moved to the front, 97 of 100 do. Four rows lose part of the streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. name in the parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates., and two land on a streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. with the same name in a neighboring commune.

The page states next to the score that the test is circular. A BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure.-backed tier graded against BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure. measures parsingaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. and routing. It does not measure coordinate accuracy.

France — 100 addresses from the BAN

I want to see what happens without one

Belgium has no rooftoprooftopGeocoding precision at the building or parcel level — coordinates within a few meters — the highest tier of the geocode cascade. Sourced from address-point and situs data. registerinput modeThe Decision-A register switch: 'fragmented' (human-typed fragments — feeds the evidence channels) vs 'formatted' (complete records — runs the trained absence identity). Explicit on CLI/API; per-endpoint defaults (batch→formatted, autocomplete→fragmented); kind-derived otherwise. and no localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. of its own. It sits in tier 2, served only by the global candidate gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture.. The panel runs thirty bilingual Belgian address lines through the global candidate gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. in three configurations. With default settings, 24 of 30 land in Belgium. With the countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. pinned, 30 of 30 do. Without the pin, six rows go wrong. Five of them fail because a countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. prior filters out the correct answer before ranking runs.

Belgium — 30 addresses, three configurations

I want to know how to read a geocoding benchmark

A geocodinggeocodingThe process of converting an address into geographic coordinates (latitude and longitude). Mailwoman geocodes in a multi-tier cascade: exact address-point match → street interpolation → locality centroid. Each tier is progressively coarser but more widely available. benchmark can look better than it is in four ways. A resolve rate can count fallbacks as successes. An evaluationevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. set can be drawn from the reference data it is graded against. A small panel can be quoted as if it represented a population. A timing figure can change on a different host. The page illustrates each case with one of our own numbers.

How to read our numbers

Run the benchmarks yourself

run-all.sh pulls the two data bundlesevidence bundleThe pair of retrieval-augmented input channels (street-type + locality-surface) that feed lexicon membership as soft per-token evidence alongside the text. Shipped in 6.7.0; trained natively from step 0 in the from-scratch base line. the panels need, the 2.88 GB global gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. and the 6.95 GB French rooftoprooftopGeocoding precision at the building or parcel level — coordinates within a few meters — the highest tier of the geocode cascade. Sourced from address-point and situs data. database. It then runs both panels in one pass:

npm install mailwoman @mailwoman/neural @mailwoman/neural-weights-en-us \
            @mailwoman/neural-weights-fr-fr @mailwoman/resolver \
            @mailwoman/resolver-wof-sqlite @mailwoman/ban @mailwoman/core @mailwoman/spatial
curl -fsSLO https://mailwoman.ai/benchmarks/run-all.sh
curl -fsSLO https://mailwoman.ai/benchmarks/fr-ban-panel.ts
curl -fsSLO https://mailwoman.ai/benchmarks/fr-ban-sample.json
curl -fsSLO https://mailwoman.ai/benchmarks/be-panel.mjs
curl -fsSLO https://mailwoman.ai/benchmarks/be-panel.json
bash run-all.sh ./benchmark-data

The Belgian panel needs only the first bundleevidence bundleThe pair of retrieval-augmented input channels (street-type + locality-surface) that feed lexicon membership as soft per-token evidence alongside the text. Shipped in 6.7.0; trained natively from step 0 in the from-scratch base line.. To skip the 6.95 GB of French rooftop datasitus dataA dataset of exact address-point coordinates (rooftop-level). Mailwoman's geocoder uses a national situs layer (124.9M US points built from state address-point sources) as the highest-precision tier of the geocode cascade., run the Belgian panel by itself with node be-panel.mjs.

Your numbers will differ from ours where the versions differ. A newer BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure. release moves rows, a newer gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. changes rankings, and a newer modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' changes parsesaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates.. Each result file has a versions block that records the Mailwoman version, the resolved modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' artifact and the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. file, so you can trace a difference to its version. The French panel also records its BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure. release on the committed sample file. The Belgian panel records no data release, because Belgium has no registerinput modeThe Decision-A register switch: 'fragmented' (human-typed fragments — feeds the evidence channels) vs 'formatted' (complete records — runs the trained absence identity). Explicit on CLI/API; per-endpoint defaults (batch→formatted, autocomplete→fragmented); kind-derived otherwise..

Our runs used Mailwoman 8.7.0, model.onnx from neural-weights-en-us@7.0.0, and candidate-global-1026.db. npm currently serves 8.6.0, so an unpinned install is one version behind these pages.

Earlier versions of these pages called that modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' file model-v401-base-step-060000-int8.onnx. The bytes are the same, with md5 c968c24ad1a3e59658ff1d92c5720575 in both cases. That name existed only on the trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. machine and reached the result files through a development symlink. The weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. package publishes the file as model.onnx, so your own re-run will record that name.

What is not here

These pages do not compare Mailwoman against another geocoder's output. To benchmark someone else's service means running it under terms we did not write, on a query mix we chose, and publishing an accuracy number for a system we do not maintain. Compare sets out what each of the alternatives is built for and when to choose it, without claiming an accuracy figure on its behalf.