DEV Community

Cover image for Lemonade Second Squeeze: Model Archeology on 2019's GPT-2XL
L. Cordero
L. Cordero Subscriber

Posted on

Lemonade Second Squeeze: Model Archeology on 2019's GPT-2XL

Two weeks ago I had never run an AI model on my own machine. Every project I had ever built phoned a cloud API with a key sitting in it. Then I sat in the vibecoding lab at AMD's Advancing AI 2026 summit and watched a model answer a question from a laptop, nothing leaving the room, and I got the itch.

I have a pattern. When something lights me up, I chase it before I am ready. This was that. I picked a project that was over my head on purpose, and then I spent a week refusing to put it down.

So before any numbers, here is the crux of it: the numbers are not the point, well kind of they are. The point is that I kept hitting walls and kept going anyway. The rest of this is just how.

I picked a project I had no business starting.

The idea was model archaeology. Take a model from 2019 and a model from 2025, run them both on the same laptop, and measure what six years actually changed. I called it Second Squeeze. Squeeze an old lemon, squeeze a new one, see if the juice is the same.

Old lemon: GPT-2 XL, 2019, 1.5 billion parameters. New lemon: Qwen3-0.6B, 2025, less than half the size. Both running locally through Lemonade, which is AMD's local AI server. I had never installed anything like this in my life. I started anyway.

The install fought me, and I kept going.

It did not go smoothly. But I'm always up for a challenge! The command was not the one the docs implied. It was not on my PATH the first time. A GPU backend tried to download and jammed on a locked file, twice. My Python was too new for the eval tool, so I stood up a second Python just for this project.

None of that is glamorous. But figuring it out is all the fun! Every wall was the same small decision: close the tab, or read the error and try one more thing. I kept picking one more thing. Eventually a tiny 2025 model said hello to me from my own laptop, and I may have made a noise, HOORAY!

My first real plan hit a wall, so I moved the wall.

The original plan was accuracy. GPT-2 has a famous 2019 benchmark, and I wanted to see if the old claim held up. That test needs a specific kind of probability data from the model, and Lemonade did not hand it back the way the standard tool expected. Dead end.

Old me would have read that as proof I was not cut out for this. Claude and new me read the docs instead, pinned down exactly what was missing, reproduced it cleanly, and filed an issue on Lemonade's repo so the next person hits a signpost instead of a wall. My first open source contribution, born entirely from something not working.

Then I changed the question to one the tool could answer. Not how smart, but how fast, and can it even follow an instruction. Same curiosity, different door.

My own benchmark caught me lying, and I let it.

This is the part I am proudest of, and it started as a mistake.

My first speed run had GPT-2 at 0.31 tokens per second. Next to the new model's 90, that is a 297x gap. Huge. I wanted it to be true so badly.

Before I posted it, Claude wrote a little script so anyone could re-run every number with one command. I ran it. GPT-2 came back at 8.5, not 0.31. My first number was junk, measured while a backend was still downloading and the model was cold off my disk. The real gap is about 11x, not 297x.

I could have kept the 297x. Nobody would have checked. Instead I kept the 11x and wrote down why the first one lied, because a number you cannot reproduce is not a result, it is a rumor. The tool I built to keep myself straight is the tool that caught me. That felt like the whole month in one moment.

And the capability side, quickly, because it made me laugh: I asked both models to finish "The capital of France is." The 2025 one said Paris and even showed a little reasoning first. The 2019 one, which has no concept of a conversation, produced a wall of random import statements. Six years is a long time.

What I kept wasn't the number.

Here is what I took from the build, and it is not tokens per second.

Every single step was something I had not done before. Running a model locally. Reading a stack trace with Claude's assistance. Filing an issue on Lemonade's repo. Saying in public that my own headline was wrong. Any one of those was a fine place to quit, and the only thing that made this a project instead of a folder of half-attempts is that I did not quit.

I think that is the skill I love strengthening, more than any framework. Keep trying the new thing. Keep pushing when it pushes back. Change the door when the door is locked, but do not leave the building.

Second Squeeze is live. There is a dashboard you can click, a repo you can re-run, tests you can try to break, and an issue I filed that might help close a real gap. Next I want to run the whole thing on an actual AMD GPU and add another bar. That is another burst, another day.

If you have been circling something that feels over your head, take this as your sign. Start it before you are ready. Hit the walls. Keep going.

For you, if you want it:

Quick context if you are new here: I came to code from the courtroom. Jury services to AI builder in about a year, self-taught, learning in public. I direct, the agents generate, I validate and decide. That is the lens I am writing from.

AI Assisted. Human Approved. Powered by NLP.

Top comments (4)

Collapse
 
kartik-nvjk profile image
Kartik N V J K

Going back and measuring 8.5 tokens per second against an advertised 297x is the step most benchmark writeups skip. A speedup number without the baseline and the harness attached is close to unusable, and an 11x actual gap is still a good result to report honestly. What did you use to keep measurement conditions consistent between the two runs?

Collapse
 
earlgreyhot1701d profile image
L. Cordero

Great question. Claude and I worked together to get this as right as possible. Plain version:

Everything ran through one tool, Lemonade's built-in bench, with identical settings every time: 5 warmup passes that don't count, then 10 timed passes, a 64-token prompt in and 32 tokens out. The warmup is what lets the machine settle before the stopwatch starts. For the two-model comparison I also forced both onto the same backend (CPU), so the hardware wasn't a hidden variable. Then I wrapped the whole thing in a small script (bench_all.py) so every run fires the exact same commands instead of me retyping them slightly differently, and the versions are pinned in an environment file so someone else can match my setup.

How I knew the conditions actually held: the newer model reproduced almost to the decimal across runs, 90.845 then 90.853. Same tool, same settings, basically the same number. That told me the setup was stable, and it was the old model's number that had really moved.

The bad 0.31 run was itself a consistency failure. Same tool, but the conditions weren't clean: a GPU backend was still downloading in the background and the 6.8GB model was cold off disk while it timed itself. It measured my laptop having a bad moment, not the model. The fix wasn't a cleverer method, it was rerunning once things settled, and the script made that trivial.

I'm not a CS or ML person, and I used Claude to help set up the benchmark parameters and to work through answers like this one. I direct and decide, the AI helps me get there. I leaned on the tool's own methodology (warmup plus repeated timed runs) rather than inventing my own. The baseline, harness, and exact commands are all in the repo if you want to poke at it.

Collapse
 
sarahpan profile image
Sarah Pan

Really interesting project! I appreciate how you act quickly and get involved when you see something interests you.

Collapse
 
earlgreyhot1701d profile image
L. Cordero

Thank you. Diving right in is half the fun and the other half is figuring things out when I get stuck! 🤣