We lost to Nominatim in Europe. Then we found out why.
Our geocoder had beaten Nominatim across the United States by fifteen points, so we ran the same benchmark on Europe and expected a similar result. Europe produced a double-digit loss.
A loss to Nominatim would have been acceptable, because Nominatim covers the whole planet on community-contributed data and is the standard to beat. We could not explain the loss, though. We knew our parser was not ten points worse in Europe than in America, so we investigated what the gap consisted of.
This post covers the cause, the fixes, and the problems the fixes did not solve. For the European comparison we added a third system: Pelias, through geocode.earth. geocode.earth hosts Pelias on Elasticsearch. Pelias turned out to be the stronger competitor, and this post states where it still beats us.
How we kept score
We graded on where the address lands on Earth, as we argue geocoders should be graded, and ignored whether the model's labels matched ours. OpenAddresses ships a truth coordinate with every row. For each address we parse it, resolve it to a point, and measure the great-circle distance to the true location. A result within 25 kilometers counts as correctly placed, and a no-result counts as a miss, the same as a wrong answer. Each system gets one number on the same real addresses, and each gets the same country hint.
This standard is stricter than the one we had been using internally. Our internal "resolve rate" counted any resolution as a success, including results that landed a region away. Re-graded at 25 km, the internal metric had overstated our European results by fifteen to twenty points. The benchmark's first effect was to correct our own scoreboard, and we now grade the coordinate.
The US, briefly — because it's why Europe stung
In the US, mailwoman correctly places 99% of these addresses at 25 km, and Nominatim places 84%. Almost all of the difference is no-result: ours is near zero, and Nominatim's is about one in six. That difference reflects our purpose-built data, with TIGER and a fifty-state rooftop layer covering areas where community OSM data is still incomplete.
We get that result from a 30 MB model that runs in your browser, without Elasticsearch or a server round-trip. The same engine, deployed the same way, ran on Europe. That is why a double-digit loss there was confusing.
Europe: we weren't wrong, we were silent
Graded the same way across six European locales, mailwoman started at about 66%. Nominatim scored 78%. Pelias uses Elasticsearch and a mix of sources, and it scored 89%. We were behind both. The obvious explanation is that the parser is worse in Europe, but the data did not support it. When mailwoman does resolve a European address, the point is within a kilometer or two of the truth. Almost the whole gap came from the other failure mode: more than one in four messy European addresses came back empty. The model did not place them wrong. It returned no place at all.
Empty results are easier to fix than wrong ones. We investigated why the results were empty and found two separate causes.
Why it went quiet, part one: a word it mispronounced
The first cause is fragmentation. The subword tokenizer splits a town like Grudziądz at the accented ą into Grudzi + dz. Neither half is a place, so neither resolves, and the whole address fails. The full town name was present in the input the entire time. The tokenizer split a word that it had received intact.
The unresolved results include Polish, Czech, and Portuguese towns that the model split at a diacritic and then could not find.
Why it went quiet, part two: postcodes we never had
The second cause was missing data. We had no postcodes for Poland and Czechia. WhosOnFirst, our gazetteer's main source, has rich postcode coverage for some countries and none for others, and PL and CZ had zero rows. A Polish address whose town the model did read correctly could still fail, because the postcode it could have resolved by was absent from the database.
The fixes: read the word, and fill the gap
For the first cause, when a parse resolves no place, the resolver now returns to the raw text, using the actual characters with diacritics intact instead of the model's subwords. It enumerates the whitespace tokens, matches each against the same-country gazetteer, and takes the longest exact match because a specific town name holds more information than an ambiguous prefix. Before trusting it, we checked that it recovered real coordinates. The recovered towns land about 1.8 km from the truth, and not in a place with the same name elsewhere in the country.
For the second cause, we imported the missing postcodes from GeoNames: about 73,000 for Poland and 15,000 for Czechia, each with its own centroid and an open license. We added them directly to the gazetteer without a model change or retrain.
We did not expect the next result. The two fixes mostly solve the same cases by different routes. Where we now have the postcode, the address resolves directly and word recovery has no work to do. Where we lack it, recovery resolves the address. Graded the same way, the postcode fill by itself raised Europe from 66% to 84%. Word recovery by itself also raised it from 66% to 84%. Together they reached 88%, and the empty-result rate fell from more than one in four to under one in twelve.
The third fix: stop being confidently wrong
At 88%, a set of misses remained, and these were wrong results instead of empty ones. "06260 Saint-Pierre, La Placette 7" came back 617 kilometers from the truth. Saint-Pierre is one of the most common town names in France, and we had picked the wrong one. The postcode, 06260, had resolved correctly to the Alpes-Maritimes, where the address is. The coordinate picker trusted the town name and never checked it against the postcode in the same parse, so it discarded the correct answer.
We added that check. When a town lands far from its own postcode, the picker chooses the town of that name nearest the postcode instead. If none is close, it falls back to the postcode's own point. This applies the consistency check from recovery to the main path, without a model change. It raised Europe from 88% to 94%. After we extended the GeoNames postcode fill to Portugal, Austria, and Australia, it also raised Australia from 35% to 65%, because a resolved postcode gave the disambiguation a reference point.
The direct part
We have been wrong before by declaring success early, so here are the numbers without interpretation. Graded the same way on the European set, all three systems side by side: mailwoman 94%, Nominatim 78%, and Pelias 89%. We now lead both, by five points over a hosted Elasticsearch stack and sixteen over Nominatim, with a 30 MB model running in a browser tab. We lead both systems in Italy, Portugal, France, and Czechia.
Across all seven locales including Australia, the results are 90% for us, 81% Nominatim, and 88% Pelias. We still lead, but the margin is thin, and Australia accounts for all of the narrowing: we score 65% there to Nominatim's 97. We have now worked out that Australia is a parsing problem. Australian addresses in this set lead with the postcode ("3053 Carlton, Barry Street 50"). Our model, trained on American and European formats where the number comes first, reads the leading 3053 as a house number and mislabels everything after it. Given the same address reordered in the American format, it parses correctly, and 65% becomes 87%. The fix belongs in the model: we will train it on the Australian format using the national address file. Australia remains unfixed. We know the cause, and it lies in the parser. The reordered addresses resolve with the existing gazetteer, so the gazetteer needs no change.
There is also a precision cost. At 25 km we're competitive, and at one kilometer we are not. We resolve to admin and postcode centroids, so we place the area correctly, within a couple of kilometers, where Nominatim and Pelias place a rooftop pin. The headline result therefore means the right neighborhood and not the right building. A neighborhood-level result answers an area-level question. A rooftop result requires building-level data.
Word recovery stays off by default, because about one in six of the newly recovered addresses lands more than 25 km away when recovery matched the wrong place with that name. Where we have postcode coverage, a consistency check rejects those before they are returned. The GeoNames fill means that check now runs for Poland and Czechia, where a month ago it could not. Australia is where it still cannot run. A batch pipeline therefore leaves recovery off, because a wrong coordinate corrupts a join without any visible error. The demo turns recovery on, because in an interactive tool a visible, correctable result is more useful than no result. Every recovered place is tagged with whether the postcode check ran, so each caller can decide how to treat it.
The thing a scoreboard can't show
A single resolve rate also leaves out confidence. A search index returns a result without saying how much to trust it. Mailwoman's confidence is calibrated: a span it marks 90% is correct about 90% of the time, measured on held-out data. You can build a rule on that number, such as auto-accepting above a threshold and routing the rest to a human reviewer, and the threshold behaves as you set it. The recovered-but-unverified coordinates above are flagged separately and are not reported as calibrated probabilities, because the calibrated number is only useful if it stays accurate.
Here is where we stand. Pelias is ahead of us across the full panel, and it places rooftops where we place neighborhoods. In Europe we beat Nominatim by a wide margin and matched Pelias, using 30 megabytes in a browser, without Elasticsearch, and with a calibrated confidence number. We still trail on parts of the map, and Australia is unsolved. We mostly closed a double-digit gap by reading town names that the tokenizer had been splitting and by adding postcodes we should have shipped from the start. We will keep grading on the coordinate.