The gazetteer prior

The question

You have a list of every place nametoponymA proper name for a geographic place. in the world, with a type attached to each. Why does the parser not read the input, check each tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. against the list, and labelcomponent tagOne of the 25 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. the hits?

The list knows which strings can be place namestoponymA proper name for a geographic place., but it does not know which strings are used as place names in a given input. This page explains how Mailwoman uses a place list with that limitation. The parser keeps the list and consults it on every parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates., but the list does not decide the labelscomponent tagOne of the 25 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag..

The analog

Most people start with a gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. lookup. They tokenize the input, probe a table of place namestoponymA proper name for a geographic place. with each tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words., and labelcomponent tagOne of the 25 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. every tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. found in the table as a place.

This approach works on a large fraction of real input, but it fails on common cases. In Park Avenue, Park is a place nametoponymA proper name for a geographic place. somewhere, so the lookup labelscomponent tagOne of the 25 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. the streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. as a town. In Paris, TX, the lookup has no reason to prefer the Texas Paris over the French one, because the table makes no statement about the TX next to it. In Buffalo Health Clinic, Buffalo, it finds Buffalo twice and cannot tell which one is the business and which is the town.

All of these failures have the same cause. The lookup answers "is this string ever a place nametoponymA proper name for a geographic place.", but the parser needs to know "is this string used as a place nametoponymA proper name for a geographic place. here". The first question is about the world, and the second is about the sentence. The table can only answer the first.

A better table would not fix this. The fix is to treat the table's answer as one piece of evidence and combine it with evidence about the sentence. The per-piece scores already provide that evidence.

The mechanism

Before the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' runs, the parser walks the input against a place index and collects every match. For each match it produces a small set of adjustments to the per-piece scores:

• A bonus on the matching component labelscomponent tagOne of the 25 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag.. A localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighborhood in the hierarchy. match adds to B-locality and I-locality on the pieces it covers.
• A penalty on the competing labelscomponent tagOne of the 25 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag.. The same match subtracts from streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels., house numberhouse numberThe numeric or alphanumeric identifier of a building on a street. Mailwoman's house_number component; its position relative to the street name flips between locales. and venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. on those same pieces. A spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. that matches a known town is evidence against those readings as well as evidence for its own.

The parser adds those adjustments to the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s scores, and the decoderdecoderIn a transformer encoder-decoder model, the part that produces output sequences. Mailwoman's classifier is encoder-only (no decoder); the 'CRF decoder' is a different thing — a structured-prediction layer that picks the best label sequence from the encoder's outputs. reads the sum. This technique is called shallow fusionshallow fusionBlending an external knowledge source (a gazetteer, an FST prior) into a model's decision as a signal, rather than retraining the model or overriding its output. Mailwoman applies it at the input layer (the gazetteer anchor as membership features) and at decode time (an FST prior biasing emissions). RAG-shaped, but the retrieval lands as feature vectors, not prompt text.. It comes from speech recognition, where a language modellanguage model (LM). A model that assigns probabilities to sequences of tokens. Used here mostly as a prior — an FST or n-gram model that biases the decoder toward plausible sequences via shallow fusion.'s scores are added to an acoustic modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s scores at decode time instead of replacing them. The combined scores retain both sources of evidence. The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' contributes what the sentence looks like, the index contributes what the world contains, and the parser consults both on every input.

The bonus is scaled by how prominent the matched place is and capped at three points of score. The cap keeps the index from overriding a confident modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'. When the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s top two readings are far apart, three points does not close the gap, and the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s reading wins. When they are close, three points decides the result. The index therefore has the most influence where the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' is least certain.

Three details explain behavior you can observe.

Match length changes the size of the penalty and leaves the bonus unchanged. Length is counted in words. A one-word match is weak evidence, because surnames, common words and streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. headsattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. are place names somewhere. A match spanning several words is stronger, because a coincidental match becomes less likely with each word. By default, the parser discounts the penalty on competing readings for a short match and keeps the bonus at full strength. A bare Hollywood is a one-word match, and the full bonus is what makes it come back as a localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighborhood in the hierarchy.. If the parser discounted the bonus, the index would lose its most important effect. A wrong one-word match does its damage through the penalty, so the penalty is the part that is reduced.

A match in a streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.-like context gets no bonus. When a matched name is in a streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.-name position, with a streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.-type word beside it or a house numberhouse numberThe numeric or alphanumeric identifier of a building on a street. Mailwoman's house_number component; its position relative to the street name flips between locales. to its left, the runtime pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-hint → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. withholds the positive bonus and keeps only the penalty. The index therefore does not pull a streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. that takes its name from a town toward the town. 800 Washington Ave, Portland, OR and 123 Madison Ave, New York, NY 10016 parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. identically with and without the index. The penalty is never withheld, because the index provides evidence for a reading and against its competitors. It never penalizes a reading it has no information about.

One spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. can match several kinds of place at once, and the index does not choose between them. New York is both a citylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighborhood in the hierarchy. and a state, and both entries apply. Each adds a bonus to its own component labelscomponent tagOne of the 25 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. on the same pieces, and the decoderdecoderIn a transformer encoder-decoder model, the part that produces output sequences. Mailwoman's classifier is encoder-only (no decoder); the 'CRF decoder' is a different thing — a structured-prediction layer that picks the best label sequence from the encoder's outputs. decides which one wins by reading the rest of the line.

What it can and cannot do

It cannot resolve. The index reports that a spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. could be a place of some type. It does not choose which place, and it produces no coordinate. The resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. does that after the parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates.. Gazetteers describes the database, and The two architectures describes where the two jobs sit.

It covers four component types. A match becomes a bonus only when its place type has a modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' labelcomponent tagOne of the 25 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag.: countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head., regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality., localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighborhood in the hierarchy. or postcodepostcodeThe country-specific postcode (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.. The index holds more types than that. A query for Boston returns neighborhood entries as well as the localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighborhood in the hierarchy.. Entries outside those four types contribute no bonus, because there is no labelcomponent tagOne of the 25 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. to apply the bonus to.

It contains administrative names only, without streetsstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. or venuesvenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label.. A gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. of specific streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. names would add several million entries, and the package does not ship one. A separate, much smaller vocabularyvocabularyThe fixed set of tokens a tokenizer can produce. Mailwoman's SentencePiece vocabulary is tens of thousands of subword pieces, with byte fallback for anything outside it. of generic streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.-type words (avenue, rue, straße) exists, but it is a different mechanism with its own wiring status. VenuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. names are open-ended and are not in any index. This matters because the hardest ambiguity in address parsingaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. is a venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. that takes its name from a place, and the index has the least information about that case.

It cannot override a confident modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'. The bonus is capped, and the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s scores are not. If the parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. is wrong in a way that a place-name lookup would fix, the index will not fix it.

It is a prior instead of a stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale check, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone)., so a missing index produces no error. The runtime pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-hint → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. loads the index from the weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. package on its first call and passes it to every parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates.. A classifier you construct and call directly receives no index. The parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. still succeeds, but it is less accurate on the inputs the index is meant for, and it breaks silently. A bare Hollywood comes back as a streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. without the index and as a localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighborhood in the hierarchy. with it. Library API has the wiring and the measured difference.

What it costs

The index is the second-largest file in the weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. package, after the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' itself. It is 21.8 MB, against 39.4 MB for the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'. Footprints lists it with the other files. The design spends build time to save query time. Every place nametoponymA proper name for a geographic place. is compiled into a walkable structure once, offline, and a parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. only walks that structure. Because of this, the same index can ship to a browser instead of running behind a lookup service.

A correction from the index does not raise the reported confidence. The score on each spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. is the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s own score for the chosen labelcomponent tagOne of the 25 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag., computed before the index adjustments are added. When the index changes a reading from wrong to right, the corrected spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. shows the score the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' gave that reading on its own. That score is low, because the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' preferred another labelcomponent tagOne of the 25 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag.. On a bare Hollywood, the score is 0.691 as a streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. without the index and 0.264 as a localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighborhood in the hierarchy. with it. The labelcomponent tagOne of the 25 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. improved, and the score fell. If you threshold on confidence, your threshold will send this kind of correct answer to review. Calibration and confidence covers how to handle it.

The index covers only what the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. covers. The prior cannot help with a town that is missing from the open data. That failure looks like an uncertain modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' instead of a missing record.

The index does not fix the venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label.-versus-place case. The parser still gets that case wrong. Riverside Garden Center, Boston comes back with two localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighborhood in the hierarchy. spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. through the pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-hint → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package.. The business name is labeled a localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighborhood in the hierarchy. across characters 0 to 22, and Boston is correctly read at 0.784. The town is labeled correctly. The parser fails to decide that the leading phrase is a venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. instead of a second place nametoponymA proper name for a geographic place.. The output is identical with and without the index, both through the pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-hint → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. and through a bare classifier call. Riverside and Boston are both in the index, but neither match changes that decision. The parser must judge how the words are used to separate a business name from the town after it, and the index has no entries for business names. A prior that supplies world knowledge without sentence structure will produce this kind of failure.

Related

• How a model reads an address — where this sits in the chain.
• Gazetteers — the place database, and what is in it.
• How the decoder chooses a path — how the decoderdecoderIn a transformer encoder-decoder model, the part that produces output sequences. Mailwoman's classifier is encoder-only (no decoder); the 'CRF decoder' is a different thing — a structured-prediction layer that picks the best label sequence from the encoder's outputs. reads the adjusted scores.
• Library API — the fst option, and what a bare classifier call misses.
• Footprints — measured file sizes, with the commands that produced them.