Getting a model onto a phone means exporting it, and almost always quantizing it. Both of those change the numbers. Everyone knows that part.
What got me is that nothing tells you when they change too much. The export succeeds. The app builds. The model runs on device and hands back something that looks completely reasonable. And the thing you shipped is worse than the thing you evaluated, by some amount nobody measured, and you find out from a support ticket a month later. Or you don't find out.
I went looking for the standard answer, assuming I'd just missed it. There is one, sort of. Compare your offline predictions against your online ones, and against the post-serialization ones. Every deployment guide says some version of that. Not one of them turns it into something a build can fail on. It's written up as a thing you should do, which in practice means you do it once, the week before launch, while you're doing forty other things, and then never again.
There's a second gap and it fails the opposite way. Preprocessing gets written twice. Once in Python, for training. Once in Dart, for serving. Same normalization constants, same resize, same channel order, sitting in two files that nobody diffs. They agree the day you write them. Then someone changes the mean and std on the Python side six weeks later and the app just quietly starts feeding the model something it's never seen.
So I built the thing I wanted. It's called Fluttorch, it hit 1.0.0 this week, and honestly the release is the least interesting part of this post.
One document, three readers
The idea is small: the exporter writes a manifest next to the artifact, and nothing on the Dart side is allowed to restate what's in it.
Shapes, dtypes, preprocessing constants, labels, the weight hash. Written once. Three things read them, the code generator, the parity gate and the runtime, and none of them declares its own copy. If the Dart side can't restate it, the Dart side can't disagree with training.
The runtime also refuses to load an artifact whose content hash doesn't match what the manifest recorded. That sounded a bit paranoid when I wrote it. Then I re-exported and updated only half the pair, and got an artifact that satisfied every shape check and returned every number wrong, and I stopped thinking it was paranoid.
The generated side
fluttorch_gen is a build_runner builder. It turns the manifest into Dart sitting next to it, and you commit what it generates so CI can regenerate and diff. That diff is what catches a manifest that moved without anyone rebuilding.
final classifier = await Classifier.load(runtime, artifact: bytes);
final out = await classifier.run(image: ClassifierImage(pixels));
final scores = out.logits.values; // Float32List, over the same memory
Three things that used to be runtime surprises are compile errors now. Passing the wrong tensor, since each one is its own extension type. Passing them in the wrong order, since run takes named arguments. Reading an output that doesn't exist.
None of that costs anything at run time. An extension type is just the underlying tensor once it's compiled. Wrong length still gets caught, at construction, and it tells you which tensor:
TensorShapeException on "image": expected 3072 values, got 100
The bit I actually care about
At export time you hand the exporter some representative inputs and it records what the model answered. Those answers replay in your test suite, against the quantized artifact, on the real backend.
test('the quantized model still agrees with the one we evaluated', () async {
final goldens = await DirectoryGoldenBundle.open(
'build/classifier/classifier.fluttorch.json',
);
final model = await runtime.load(
artifact: await File('build/classifier/classifier.pte').readAsBytes(),
manifest: goldens.manifest,
);
await expectParity(model, goldens: goldens);
});
The tolerance comes from the recipe and the precision the manifest recorded, so an int8-dynamic model isn't judged against the bound a full-precision one answers to. You can pass your own, and you probably should if your activation ranges look nothing like the ones the defaults came from.
When it fails it looks like this:
FAIL parity/case-3
backend: xnnpack quantization: int8-static
output "load_mw" max |Δ| 1.72 > Tolerance(atol 0.1, rtol 0.1, cos ≥ 0.998)
worst at [0]: 14.0210 vs 12.3000
2 of 4 elements (50.0%) exceed the elementwise bound
no layer attribution: backend "xnnpack" offers no activation taps
Which tensor drifted, which bound broke, which element broke it, which backend produced the number. I put all four in because I kept hitting failures where I had three of them and was still guessing. That last line is there because per-layer attribution needs the backend to expose intermediate activations and xnnpack won't, so instead of silently giving you less, it says so.
The thing I got wrong
I measured the default tolerances instead of picking them, mostly out of stubbornness, and the measurement told me something I'd have got backwards.
The two-layer model drifts up to eight times further than the convolutional one. The simpler network. I stared at that for a while assuming I'd broken the harness.
It isn't complexity. The two-layer model's outputs land around 9.4, the other ends in a softmax, and relative error is measured against the output while the rounding happened on intermediates. Big outputs absorb the same underlying error inside a much smaller relative bound. Obvious once you see it, and I did not see it for an embarrassingly long time.
One bound in that table is still unmeasured and says so, because nothing in the suite exports int4 yet and I'd rather have a hole than a plausible-looking number.
There's also a model that ExecuTorch happily lowers and then can't execute, identically under its own Python runtime. I left it in the suite as a failing expectation rather than skipping it, so whenever upstream fixes it, my tests are what tell me.
What it doesn't do
It doesn't train anything, convert between formats, or serve inference. It takes something you exported with torch.export and makes the border between Python and Dart typed and checked, and that's the whole scope.
ExecuTorch was just the first backend, not the shape of the thing. ONNX Runtime and LiteRT sit behind the same seam, and nothing above it knows which one is running.
Where it is
fluttorch and fluttorch_gen are on pub.dev, Apache-2.0. The native bindings aren't published because pub.dev can't carry the native half, so you pull those from the repo.
- Repository: https://github.com/NaCode-Studios/Fluttorch
- Documentation: https://nacode-studios.github.io/Fluttorch/
If you ship models to phones I'd genuinely like to know how you handle this, especially if the answer is "I don't". That's the part I'm least confident I've got right, and I'd rather hear it now than after someone builds on it.
Top comments (2)
The dual-preprocessing gap is the one that quietly ruins the most models, and it's rarely the quantization itself — it's that normalization constants live in two files in two languages that nobody diffs, and they only have to agree once to pass review. Making the Dart side unable to restate the manifest is the right structural fix: you can't drift from a value you're not allowed to redeclare.
Where I'd extend it: the manifest + content-hash catches "the artifact changed," but not "the artifact is numerically worse within tolerance." Those are different failures. The hash gate is binary; quality regression is a distribution. A parity gate that a build can fail on needs a golden set — a frozen batch of inputs with expected outputs — run post-export and post-quantization, failing the build if top-1 agreement or max abs error against the pre-quant model crosses a threshold. That turns "compare offline vs online, once, the week before launch" into something CI enforces every commit, which is exactly the never-again-after-launch trap you named. Is Fluttorch's parity gate doing pass/fail on a tolerance, or just reporting the delta?
Pass/fail, and it fails the build. The gate replays a frozen set of goldens
captured from the source model before lowering, so what it measures against is
the pre-quant reference, and the check is an absolute plus relative bound per
output tensor with an optional cosine floor. The default bound depends on the
quantization recipe and the precision, and it was measured rather than picked. If
you hand it a recipe nobody has measured it refuses to run instead of inventing a
number, and an empty bundle fails too, because a gate that passes with nothing to
check is the exact thing it exists to prevent.
You're right about what's missing, though. Everything it measures is elementwise,
so a classifier can sit comfortably inside the tolerance and still flip its argmax
on a borderline input, and the gate would call that green. Top-1 agreement isn't a
stricter version of what's there, it's a different measurement, and I hadn't
written it. It's open now as
github.com/NaCode-Studios/Fluttorc..., with your comment linked:
the ordering-versus-distance framing is the part that made it obvious.
Two things I'd rather say than have you find. When the recipe is static, the
exporter currently calibrates on the same goldens it later grades itself against,
which makes the drift look better than it is. And the on-device half doesn't run
in CI yet: the Dart side does, against a committed export, but the workflow in the
docs is the shape it takes, not something a device proves on every commit.