Our model worked in CI but broke on every real device
We shipped a browser-based address parser that runs a 29 MB ONNX model entirely client-side. The Playwright tests showed perfect results. Chrome desktop looked great. Then someone opened it on an iPhone.
What we saw
Every address component was classified as "locality" with 0.2–0.4 confidence. "400 Broad St, Seattle, WA 98109" became three locality spans without a street, region, or postcode. The model was producing near-uniform logits, as an untrained model would.
The debug UI produced perfect results after selecting the WASM backend, with the same model bytes, tokenizer, and input. The fault came from the GPU path.
The wrong hypotheses
We spent hours on each of these before finding the real cause:
Stale browser cache. We'd recently updated the model from 25 MB (old tokenizer) to 29 MB (new tokenizer). The old model with the wrong tokenizer produces exactly this symptom. We added cache-busting query params, migrated assets to a CDN, and verified file sizes. The files were correct.
Tokenizer mismatch. The v0.5.3 model uses a 48K-vocab tokenizer but an older 24K-vocab tokenizer was briefly deployed. We verified hashes. The tokenizer was correct.
Model version drift. We have four model versions on the CDN, and the page might have loaded the wrong one. We added a version selector to the demo page and confirmed v0.5.3 was selected. The model was correct.
Browser-specific WASM numerics. Safari's WASM implementation might handle int8 quantization differently. We tested WASM on Safari, and it worked perfectly. The problem was specific to WebGPU.
Why Playwright couldn't catch it
Every automated test we ran passed, because headless Chromium does not have a WebGPU adapter. When you request executionProviders: ["webgpu", "wasm"], the runtime silently falls back to WASM. WASM handles int8 correctly, so the test passes.
We had a verify skill that launched a real headless browser, navigated to the live demo, typed an address, and checked the parse output. It ran after every deployment and passed every time. It could not catch this bug, because it never exercised the broken code path.
The cause
onnxruntime-web ships two WebGPU execution providers in the same npm package:
onnxruntime-web → ort.bundle.min.mjs → JSEP (old, broken)
onnxruntime-web/webgpu → ort.webgpu.bundle.min.mjs → Native EP (fixed)
The JSEP (JavaScript-based execution provider) has a Slice kernel bug that produces incorrect results when reversing a tensor on a specific axis. This corrupts the dequantization of int8 weights. The bug is worse on Safari's Metal backend than on Chrome's Dawn/Vulkan backend, and in our case Chrome's output happened to look correct.
The native WebGPU EP handles the same operations correctly on all backends.
The fix
- import * as ort from "onnxruntime-web"
+ import * as ort from "onnxruntime-web/webgpu"
The fix changes one line, and the API is identical. Session creation, tensor I/O, and provider fallback all work the same way. The native bundle is also smaller (113 KB vs 405 KB).
After this change, the model produces correct results on Chrome, Safari macOS, and iOS Safari. All three run on WebGPU without needing the WASM fallback.
What we should have done differently
This diagnostic sequence would have saved hours:
- Force WASM. If results become correct, the problem is GPU-side.
- Check which execution provider is active. We lacked this instrumentation at the time and have since added a backend indicator to the demo page.
- Check the import path.
grep "onnxruntime-web"in your source. If you're importing the bare package, you're on the JSEP. - Test on Safari. If it fails on Safari but works on Chrome, the JSEP is the prime suspect.
Test infrastructure without GPU access cannot catch GPU-specific bugs. Headless browsers do not exercise hardware acceleration the way real browsers do. If your product runs on GPUs, you need at least one test that exercises the GPU path on a device that has one.