Cross-locale queries

The question

What happens when the language, script, and countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. of an address do not line up?

An American might type a Japanese address in English. A Japanese user might write a United States place nametoponymA proper name for a geographic place. in kana. Someone in India might enter a Japanese address in Devanagari. These look like one localization problem from the search box, but they ask the system to do three different jobs:

1. Read the structure of the string the user supplied.
2. Connect a written place nametoponymA proper name for a geographic place. to an entity in the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture..
3. Translate or transliterate one writing into another.

Mailwoman does the first two where its modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' and atlas have coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present.. It does not currently translate arbitrary address text.

Parsing keeps the original text

The parser labelscomponent tagOne of the 25 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree.. It might decide that Tokyo is a localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighborhood in the hierarchy. and 1-1 Marunouchi contains a house numberhouse numberThe numeric or alphanumeric identifier of a building on a street. Mailwoman's house_number component; its position relative to the street name flips between locales. and streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.-like material, but it does not rewrite either spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. as 東京 or 丸の内. The output retains the user's text and character offsets into that text.

This matters because a decoderdecoderIn a transformer encoder-decoder model, the part that produces output sequences. Mailwoman's classifier is encoder-only (no decoder); the 'CRF decoder' is a different thing — a structured-prediction layer that picks the best label sequence from the encoder's outputs. is answering a structural question: which part of the input plays which postal role? Translation would be a second prediction with its own uncertainty, and replacing the original surface would hide that uncertainty from the caller.

TrainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. contributes to this structural competence. The corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. contains localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for.-specific component orders, Unicode scripts, mixed-script examples, and transliterationtransliterationConverting a name from one writing system to another while preserving pronunciation (Cyrillic → Latin, for instance). Needed for multilingual address handling and corpus synthesis.-shaped examples where we have defensible labelscomponent tagOne of the 25 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag.. The tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. can represent arbitrary Unicode through byte fallbackbyte fallbackA tokenization strategy where characters not seen during training are encoded as raw UTF-8 byte sequences rather than mapped to an unknown token. Mailwoman's tokenizer uses byte fallback so non-Latin scripts (CJK, Cyrillic) produce real token sequences even though training data was Latin-dominant.. Representationhidden stateThe model's internal vector for a token after the encoder has mixed in surrounding context. The contextualized representation the classifier head reads to assign a label. alone is not comprehension, though: a script absent from trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. can still produce a well-formed but wrong parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates..

The atlas supplies the bridge between names

After parsingaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates., the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. searches a gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture.. One place record may carry a local name, an English name, other language names, abbreviations, colloquial forms, and established transliterationstransliterationConverting a name from one writing system to another while preserving pronunciation (Cyrillic → Latin, for instance). Needed for multilingual address handling and corpus synthesis.:

Tokyo ─┐
東京  ──┼──> one place identity ──> coordinates and hierarchy
Tōkyō ─┘

When those surfaces are indexed, differently written queries can converge on the same stable place identifier. Mailwoman has not translated the query in this case. It has retrieved an entity through one of that entity's recorded names.

CoveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. follows the source data. Administrative places have much better multilingual alias coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. than streetsstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. and individual addresses. A gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. may know ニューヨーク as a name for New York while having no kana alias for a particular streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. in Ohio. The first can resolve and the second can fail, even though the parser successfully labelscomponent tagOne of the 25 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. both strings.

What the three examples do today

A Japanese address typed in English

The Latin-script input can be parsed as written, and established English names or romanizations can resolve when the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. contains them. Japanese support is currently resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon.-route only, with no general parser claim, so a successful administrative-name or postcodepostcodeThe country-specific postcode (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. lookup should not be read as full Japanese streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.-address coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present..

A United States address typed in kana

The tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. accepts the text and the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' may recover some structure. Resolution then depends on the atlas containing the kana surface. Most United States streetsstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. do not carry kana aliases, and Mailwoman does not silently turn the query into English. An absent alias therefore remains an unresolved query.

A Japanese address typed in an Indian script

The same boundary applies. A recorded Devanagari name or transliterationtransliterationConverting a name from one writing system to another while preserving pronunciation (Cyrillic → Latin, for instance). Needed for multilingual address handling and corpus synthesis. can reach the entity; an unrecorded rendering cannot. The coarse countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' has explicit outlier exposure for unsupported scripts so it can abstain instead of confidently assigning them to a countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. it knows.

Locale is a preference, not the address's nationality

The request localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. tells Mailwoman how the user is likely to write and interpret ambiguous terms. It does not assert that the address is in that localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for.. Input evidence such as script, postcodepostcodeThe country-specific postcode (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. shape, place relationships, and resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. candidates may point elsewhere.

When no request localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. is supplied, the runtime can use MW_LOCALE or the machine's Intl localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. as low-priority preference evidence. It keeps the machine timezone separate and reports both in the LocaleHint provenance. A server can disable host inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece. with machinePreferences: false, because a container's localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. describes the container rather than the person making the request.

If translation is added later

A translation or transliterationtransliterationConverting a name from one writing system to another while preserving pronunciation (Cyrillic → Latin, for instance). Needed for multilingual address handling and corpus synthesis. service should propose additional lookup surfaces while preserving the input:

{
	"original": "ニューヨーク",
	"candidate": "New York",
	"mechanism": "transliteration",
	"provider": "example-provider"
}

The resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. could score that candidate beside recorded aliases. The result should expose how the candidate was produced, and failure should remain visible. That is especially important for streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. names, personal names, homophones, and languages with several competing romanization systemsexpectation-maximizationAn iterative algorithm that estimates model parameters when some variables are unobserved. In Mailwoman's matcher, EM learns the Fellegi-Sunter m and u parameters from unlabeled data — no training labels needed..

The current boundary

Mailwoman can parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. multiple scripts and resolve cross-language names when its atlas contains the corresponding aliases or transliterationstransliterationConverting a name from one writing system to another while preserving pronunciation (Cyrillic → Latin, for instance). Needed for multilingual address handling and corpus synthesis.. It does not translate arbitrary addresses. Unsupported cross-script queries may still tokenize, but they should fail or abstain rather than acquire an invented mapping.

Related

• Locales and tiers — measured parser and resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present..
• Gazetteers — where multilingual place namestoponymA proper name for a geographic place. live.
• Training and the corpus — what the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' learns from data.
• What the model cannot do — script and tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. limits.