We sign containers, packages and commits. Model weights we take on trust.
You downloaded a model last week. What did you actually get?
You got a file. It came from a hub, or a mirror of a hub, or a helpful colleague who re-uploaded it somewhere faster. You checked the file size. Possibly. Then you loaded it and started serving traffic.
I do not think anyone is being careless. I think the tooling simply never asked the question, so nobody got in the habit of answering it.
Everything else in the pipeline is signed
Look at what a normal deployment already insists on.
Container images have digests, and you pin them. Packages have signatures and attestations, and your build fails when they do not verify. Commits get signed, and the review UI puts a badge on them. Every one of those was a fight at some point, and every one of those fights is now settled. Nobody argues that a container should be pulled by mutable tag in production.
Then the same pipeline pulls twenty gigabytes of weights over HTTPS and loads them into a GPU with no signature, no publisher identity, and no record of what was expected.
HTTPS tells you the bytes arrived unmodified from whoever served them. It does not tell you who built them, whether the thing you got is the thing that was trained, or whether the checkpoint in front of you is the one that was evaluated for safety before release.
The weights are the artifact that decides what your product says to customers. They are the least verified thing you ship.
Why this is worse than it sounds
Three things make weights a nastier case than a package.
First, you cannot read them. A reviewer can skim a diff and notice something odd. Nobody is eyeballing 2.8 trillion parameters. There is a line in our demo output that I keep coming back to:
no human reads 2.8T parameters; the hash does the reading.
That is the whole argument. Verification of an artifact this size is not a human activity, so it has to be a mechanical one, or it does not happen at all.
Second, tampering does not look like breakage. A poisoned checkpoint loads fine, serves fine, and scores fine on the benchmarks you happen to run. There is no crash to alert you. Model behaviour is diffuse enough that a targeted change can sit undetected for a long time.
Third, the distribution path is long and informal. Weights get mirrored for bandwidth, converted between formats, quantised by third parties, and re-uploaded by enthusiasts. Every one of those steps is a place where the artifact changes hands, and almost none of them carry provenance forward.
What a manifest actually does
The fix is not exciting. That is a feature.
A Weight Custody Manifest is a small signed document that travels with a checkpoint. It states the exact weights_hash of the artifact, who built it, and what they are asserting about it. It is signed, so you can check the assertion came from the party you think it did.
That gives you the thing you did not have before: an expectation. Before the weights load, you compare what you downloaded against what the builder signed. Same hash, proceed. Different hash, stop.
Not “log a warning”. Stop, before the file is loaded, because after loading is too late to be interesting.
If a fine-tune is involved, the derivative gets its own manifest that points back at the base it came from, so the chain of custody resolves instead of dead-ending at a filename.
Run it
I would rather you did not take my word for any of this. It runs on a laptop in about a minute, with no confidential-computing hardware, no cloud account, and no signup.
pip install weight-custody-manifest
git clone https://github.com/agentrust-io/demos && cd demos
python demo.py 6
What you get is real output, not a slideshow:
weights_hash bound : sha256:99b3e4e7...3a39f9
manifest signature : True (jointly signed builder + custodian)
gate released key : True
certified hash : sha256:99b3e4e7...3a39f9
downloaded hash : sha256:b3a36b54...b9758e
matches manifest : False -> REFUSE to load
derivative : lineage verified True depth 1 root is a base: True
The third block is the one that matters. A tampered checkpoint is refused on the hash, before load, without anyone reading anything.
What this does and does not give you
Worth being precise, because precision is the point of the exercise.
This gives you integrity and provenance for open weights. Hashing and signatures are old, boring, well understood cryptography, and they do exactly what they claim here. If the artifact changed, you find out.
What it does not give you is protection against an adversary who physically owns the machine. That is a different problem with a different answer, and anyone who tells you a hardware boundary solves it completely is overselling. But that is a separate post, and it is not the problem most teams have today. Most teams have the first problem: they cannot say what they loaded.
Where to look
The demos are open source and there are nine of them, four on weight custody and five on governing what an agent does at the tool boundary. The SDK is on PyPI.
weight-custody-manifest on PyPI
If you think the model is wrong, or the threat model is wrong, I would genuinely like to hear it. Tell me where it breaks.
Top comments (0)