I promised that every number on this blog would come with how I measured it. So here's the honest scorecard for StateFrame as it ships: what I've measured, what still breaks, how I test ideas, and the stuff I believe but can't prove yet.
I'm keeping those buckets separate on purpose. Engineers can smell a blended number from a mile away, and they should.
#What I've measured
The big one is the whole reason StateFrame exists: the prompt doesn't grow.
- Largest prompt in a 300-request run
- 32,159
- Fixed budget of 33,344 tokens
- Clean requests in a row on the shipped package
- 43
- No step cap, 0 engine errors
- Agents at once on one 48 GB GPU
- 4 of 4
- Memory held at about 26.4 GB
- Hidden tests passed on a 7-file feature
- 17 of 18
- Local open model
A few more, with their conditions:
- The budget holds. On the shipped package, a 43-request run with no step cap stayed under budget the whole way. The biggest prompt was 32,032 tokens against the 33,344 budget, with zero engine errors.
- Running agents side by side works, with a cost. Four sessions ran at once on one 48 GB GPU. All four passed and memory sat at about 26.4 GB. Each one ran about twice as slow as it would alone, but together they got about 1.9 times the work done.
- Real features on an open model. A 7-file feature passed all of its protected tests and 17 of 18 hidden edge-case tests, built by an open model running on one rented cloud GPU.
- Memory stays flat. In a 26-request run the engine sat at about 1.7 to 1.8 GB, growing only a few megabytes per request instead of climbing with every step.
- It runs on Linux. On Ubuntu 22.04 it installs, and the included demo bug goes from failing to passing in 4 requests. That's one run, so call it a smoke test.
Unless I say otherwise, these are measured with one open model, Qwen3.6-35B-A3B, self-hosted on a rented cloud GPU. The StateFrame page has the full conditions for every number.
#What it can do today (and what it can't yet)
Here's what the model actually gets to work with:
- Find and read code. Search, list files, read files, and recall something it saw earlier in the run.
- Change code. Targeted edits, string replacements, inserts, new files, or swapping out a whole module.
- Check its work. Run the tests, try a quick code probe in a sandbox, write and run its own checks, and read the results.
- Finish. "Done" only counts once the checks pass, and the engine makes that call, not the model.
Around all that there's a repo map that understands 17 languages, a chat mode where you can look at the diff, undo it or apply it, crash-proof resume, and a way to see exactly what was sent on any request.
And here's what it can't do yet:
- No general shell. It runs tests and sandboxed probes, but it can't run arbitrary commands like installing packages or running build scripts.
- No plug-in tools. Adding a tool today means editing engine code. There's a guide for it, but it should be easier.
- No web or docs lookup. That's on purpose for now, since it's built to stay local.
- No deleting or renaming files, and no git beyond the diff and apply in chat mode.
Adding tools is one of the next things I'm working on: a proper way to plug tools in, and a shell that's safe to hand an agent.
#What still breaks
I'd rather you hear this from me.
- Long, hard tasks can stall. On a long, difficult run, StateFrame can decide it isn't making progress and stop itself. In one run it did exactly that at request 49. It didn't crash and the prompt was still under budget. It just knew it was stuck. That beats spinning forever, but it isn't a finished task either. Getting long, hard tasks to stop stalling is right at the top of my list.
- Big changes are still hard. Changes that touch around ten or more files, and tasks past roughly 25 model turns, can stop short. The partial work is saved.
- Prompt caching takes a hit. Because every request is rebuilt fresh, the model server can reuse less of the previous prompt than it would with a chat history. On a real multi-step task, about 25% of prompt tokens got reused.
#DeepSeek V4 Flash
I want StateFrame to work well on more than one model, so I've been testing it on DeepSeek V4 Flash too. Here's the straight version.
On an earlier build, Flash passed a standard coding task with all 14 hidden tests, 2 runs out of 2. On a harder task it kept getting stuck rereading the same files. In one stretch it went 7 runs in a row without making a single edit. The cause was on my side: when the engine trimmed a request to fit the budget, it was dropping files the model had already read, so the model went back and read them again. I fixed that. On the release build, Flash solved the harder task in 1 of 2 runs, graded by hidden tests, and 2 of 2 on the build just before it. On a larger task it still gets stuck reading instead of editing, and that's next on my list.
Two runs is a small sample, not a success rate. I'll publish more as the runs come in.
#How I test ideas fast
This is the part I think is actually interesting.
Frame splicing. Every StateFrame run is recorded, request by request, so I don't have to rerun a whole task to test an idea. I take one exact moment from a real run, freeze everything the model saw, change one thing, and ask the model again, dozens of times. Every idea runs three ways:
- the change itself,
- the original moment, untouched,
- a placebo: a change of the same size that carries no real information.
An idea only counts if it beats both. The placebo matters more than you'd think. Plenty of "improvements" turn out to work just because the prompt got a little different.
A couple that passed:
- Keep the top of the file being edited in view. The model made its edit within 2 calls 84% of the time, against 74% for the original and 73% for the placebo, over 448 trials on one task. It also reread the top of the file about half as often.
- Tell the model plainly when all the checks already pass. It finished cleanly 22 times out of 24, against 8 for the original and 12 for the placebo, and it won on all 4 tasks where I could measure it.
And the ones that didn't, which honestly matter more:
- One idea looked great for caching, but it made the model quit early. It finished 4 times out of 24 against the placebo's 21. Rejected.
- Another won on a simplified test setup and then lost badly on the real thing, 1 out of 24 against 16 for the original. Dropped.
- I even withdrew one of my own wins. It looked big, then I found the test request had been built slightly wrong, and on the corrected setup it didn't hold up. So it's gone.
Faking a mechanism before building it. Before I build something, I hand-write exactly what it would put in front of the model and splice that in. Pretend I already have a perfect version of the feature: does the model act any differently? If it doesn't, the real feature won't help either, and I skip it.
Proving it's plugged in. This one sounds dumb until it bites you. I check every feature switch against real runs to prove each one actually changed something. On one older build, 5 switches were turned on and doing nothing at all: built, never plugged in. Now a feature that's switched on isn't a feature to me until I've seen it change a real request.
Most ideas fail. That's fine. Killing a good-sounding idea in an afternoon, before it ships, is the whole point.
#The belief that got me here
One idea carried me through building StateFrame, so I want to say it out loud.
Past a certain point, the harness matters more than the model.
Don't get me wrong. I love the big frontier models. I use Claude Opus and Fable, and Astra over on ChatGPT, pretty much every single day, religiously. For the hardest problems, nothing beats a giant model. But most day-to-day work isn't the hardest problem. Once a model is good enough, and once the work is broken down far enough, what decides whether the job gets done is everything around the model: what it sees, how its work gets checked, and how the task stays on track. That's the harness, and that's what StateFrame is.
#What I believe but haven't proven
These are theories. No data behind them yet, and I'm saying so plainly.
- Tiny models can do real work if the work is small enough. I think a great ~2B model, something like MiniCPM5-2B, can handle a lot of everyday Python if the task gets broken into small enough steps. Breaking tasks down like that is the next big thing I'm building, along with pieces that hold small models up right where they tend to fall apart. The rule for every one of those pieces: it has to be neutral or a net win for bigger models too, never a crutch that slows them down. Eventually I want all of it to be tunable. No runs yet.
- A 24 GB card can run several agents. I measured 4 agents on 48 GB at about 26.4 GB. I think the same model, with its experts offloaded to system RAM, could run at least 4 on a 24 GB card, maybe even 8, as long as you can live with each one running slower. That's what I think matters to the most people, and I haven't tested it yet.
- Way more control over the model. The engine builds every single request. It decides what the model sees, which tools it gets, and what counts as done. I think that gives you far more control over how a model behaves than prompting a chat ever could.
- Behavior you tune in code. I think a lot of what we call reasoning can be written down as code, and that's a big deal for smaller models. Picture a row of switches. One setting makes an agent work like a careful senior programmer. Flip a few and the same engine becomes a quicker content writer with a better creative process. Building an agent turns into flipping the right switches and turning the right knobs until you get exactly the agent you want, every time, defined in code instead of hoped for in a prompt.
- Fewer false refusals on legitimate work. Models sometimes refuse perfectly normal jobs because a big request looks scary out of context, like editing login code or writing a security test. When the work arrives as small, clear steps, each step is easier for a model to judge on what it actually is. I expect that to mean fewer refusals on work that was always fine to do.
- More consistent behavior across models. Because the engine owns the rails and not the model, I think different models will end up behaving a lot more alike inside it. Swap the model and the work should look familiar.
- Hot-swapping models mid-task. Every request already stands on its own, so nothing ties a run to one model's history. That opens up something I think is really cool: picking the model for the step in front of it. Writing code goes to the big brain. Reading and finding your way around a codebase goes to something smaller and cheaper.
- A prompt that never grows should pay off end to end. What I've measured is that the prompt stays bounded. Whether that makes a whole task cheaper than a chat-history agent is still an open question.
On that last one: I haven't published a head-to-head against a chat-history agent, and I'm not going to pretend the answer will be one-sided. I'd bet there are jobs where a plain chat history wins, and someday a hybrid of history and state might be worth exploring too. The work I care about is the long kind, where I believe state pulls ahead. When I have numbers on that, good or bad, they'll go here.
#Where this goes
The shape is right. The prompt stays flat, the work gets checked, every request is on the record, and I can test ideas in minutes instead of days. Now it's about stacking small wins until the long tasks stop stalling.
If you run StateFrame and get a different number than mine, I want to see it. Open an issue on GitHub with your setup and what happened.