Handle messy input

Outcome. You know which input defects the parser absorbs on its own. You know the defects that the normalize stage removes before the model sees them, and when to call that stage yourself.

For most callers the answer is short. If you use geocodeAddress, preprocessing already ran. If you call classifier.parse directly, it did not. This page explains the difference.

Prerequisites

The library install from Install and first parse. Add @mailwoman/normalize if you want to call the stage directly.
You do not need a data root. No step on this page needs a gazetteer.

What runs where

The two entry points preprocess input differently:

Entry point	Deterministic preprocessing	Case normalization
`geocodeAddress(raw, deps)`	Yes — `normalize()` with abbreviation expansion, `normalizeInput: false` opts out	Yes
`classifier.parse(raw)`	No	Yes

Case normalization runs in both, because it is part of the classifier instead of the normalize stage. The other steps (Unicode form, punctuation, whitespace, abbreviation expansion) run only on the geocode path.

1. Know what the parser absorbs unaided

Before adding a preprocessing step, check whether the defect matters. Feed the model the same address in three casings:

node case-compare.mjs

input   1275 Pennsylvania Ave NW, Washington, DC 20004
  on    region="DC" locality="Washington" street="Pennsylvania Ave NW" house_number="1275" postcode="20004"
  off   region="DC" locality="Washington" street="Pennsylvania Ave NW" house_number="1275" postcode="20004"
input   1275 pennsylvania ave nw, washington, dc 20004
  on    region="DC" locality="Washington" street="Pennsylvania Ave NW" house_number="1275" postcode="20004"
  off   region="dc" locality="washington" street="pennsylvania" house_number="1275" street_suffix="ave" street="nw" postcode="20004"
input   1275 PENNSYLVANIA AVE NW, WASHINGTON, DC 20004
  on    region="DC" locality="Washington" street="Pennsylvania Ave NW" house_number="1275" postcode="20004"
  off   region="DC" locality="WASHINGTON" street="PENNSYLVANIA AVE NW" house_number="1275" postcode="20004"

on is the default. off is classifier.parse(raw, { caseNormalization: "preserve" }), and it shows what the model does with the input casing unchanged.

Look at the all-lowercase row. With case normalization off, pennsylvania ave nw splits into two street spans with a street_suffix between them. With it on, the same input returns one street. Real users type in lowercase, and a map search box receives lowercase input constantly, so the fix is in the shipped path instead of in your code.

One function handles both directions. normalizeInputCase (neural/case-normalize.ts) detects a pure-ASCII all-caps input and title-cases each run of three or more letters. It leaves one- and two-letter runs in capitals, so DC stays DC instead of becoming Dc and being labeled as a locality. For a pure-ASCII all-lowercase input it does the reverse: it title-cases the long runs and uppercases the short ones, so dc becomes DC. Both rewrites preserve length, and mixed-case or non-Latin input passes through byte-identical.

2. Read offsets against your string rather than against the tree's

Case normalization has one consequence that can surprise you:

node offsets.mjs

tree.raw: "1275 Pennsylvania Ave NW, Washington, DC 20004"
  region        value="DC"                    raw.slice(38,40)="dc"
  locality      value="Washington"            raw.slice(26,36)="washington"
  street        value="Pennsylvania Ave NW"   raw.slice(5,24)="pennsylvania ave nw"
  house_number  value="1275"                  raw.slice(0,4)="1275"
  postcode      value="20004"                 raw.slice(41,46)="20004"

The input to that run was all lowercase. tree.raw and every value come back in the normalized casing instead of the casing you sent. Because the rewrite preserves length, start and end still index your original string. yourInput.slice(node.start, node.end) returns the user's original text whenever you need to echo it.

3. Call the normalize stage when you parse directly

@mailwoman/normalize is Stage 1. It consists of pure functions and uses neither a model nor I/O. Four transforms run every time (Unicode NFC, a CJK fold, punctuation and whitespace), and each one adds a record only when it changed something. Two more transforms are opt-in.

Save this as normalize-demo.mjs:

import { normalize } from "@mailwoman/normalize"

const messy = "  1275   Pennsylvania Ave  NW , Washington，DC 20004  "
const out = normalize(messy, { expandAbbreviations: true, locale: "und" })

console.log("raw:        ", JSON.stringify(messy))
console.log("normalized: ", JSON.stringify(out.normalized))
console.log("transforms: ", JSON.stringify(out.transforms))

The transforms array records what changed, so log it when a parse surprises you. Each entry identifies the step and its effect.

This table shows what each step does, with one input per row:

Input	Normalized	Step
`a b`	`a b`	`collapse_whitespace`
`O’Brien St`	`O'Brien St`	`normalize_punctuation`
`1275 Pennsylvania Ave NW—Washington, DC`	`1275 Pennsylvania Ave NW-Washington, DC`	`normalize_punctuation`
`１２７５ Pennsylvania Ave`	`1275 Pennsylvania Ave`	`normalize_cjk`
`〒150-0001 東京都`	`150-0001 東京都`	`normalize_cjk`

The CJK step also affects non-CJK text. It folds full-width ASCII, such as the full-width digits a Japanese form produces. It also strips the postal mark 〒, which would otherwise tokenize as an unknown byte sequence in front of the postcode.

Two options are off by default. expandAbbreviations with locale: "und" applies the safe set for an unknown locale. That set contains abbreviations that mean the same thing everywhere:

"12 Bd Saint-Germain, 75005 Paris" -> "12 Boulevard Saint-Germain, 75005 Paris"  Bd->Boulevard
"3 Imp des Lilas, Lyon"            -> "3 Impasse des Lilas, Lyon"                Imp->Impasse
"1275 Pennsylvania Ave NW, Washington DC" -> unchanged

The stage leaves Ave unexpanded because the model already reads it correctly. The set targets forms the model finds ambiguous. caseFold is the other opt-in, and you should leave it off. The classifier's own case handling is more precise, and folding first removes the case information in proper nouns and directionals.

Verify

Run the stage on a deliberately messy string and parse both forms to see what the stage changed:

node normalize-demo.mjs

raw:         "  1275   Pennsylvania Ave  NW , Washington，DC 20004  "
normalized:  "1275 Pennsylvania Ave NW , Washington,DC 20004"
transforms:  [{"kind":"nfc","changed":false},{"kind":"normalize_cjk","folded":1,"stripped":0},{"kind":"collapse_whitespace","runs":4}]
raw         region="DC" locality="Washington" street="Pennsylvania Ave  NW" house_number="1275" postcode="20004"
normalized  region="DC" locality="Washington" street="Pennsylvania Ave NW" house_number="1275" postcode="20004"

Compare the last two lines. Both parses return the same five components, so the stage did not change what the model found. The only difference is one value: the raw parse keeps the double space in street, and the normalized parse removes it. For this address, preprocessing does not change the parse. It makes the extracted value clean enough to store. If those values go into a database column or a match key, run the stage first.

Limits

Zero-width characters pass through. collapseWhitespace does not count U+200B as whitespace, so a U+200B in the middle of the string passes through the stage unchanged. The parse handled one in testing, but a match key built from the result will contain it. Strip zero-width and bidirectional control characters yourself if your source can produce them.
Offsets are reliable only against the string you passed to that call. For this reason, normalize returns an offsetMap that maps normalized positions back to raw positions. If you normalize and then parse, map offsets through it instead of assuming the indexes line up.
Case normalization applies to pure-ASCII input only. Accented and non-Latin input keeps its case, because changing its case can change its length (ß becomes SS) and break the offset guarantee described above. An all-caps French address does not get the all-caps fix.
Abbreviation expansion uses a small safe set at locale: "und". A real locale widens the set. It is not a general address-standardization pass, and it does not implement USPS Publication 28.

Validate an address before you use it — checking the parse once the input is clean.
Understand a parse — what the components and offsets mean.
Tune confidence thresholds — what a low score means when the input is the problem.

Prerequisites​

What runs where​

1. Know what the parser absorbs unaided​

2. Read offsets against your string rather than against the tree's​

3. Call the normalize stage when you parse directly​

Verify​

Limits​

Related​