Fifty StateFrame releases in four days, and a server we beat like it owes us money

Four days ago I turned on a server from 2012 in my basement. I did not go easy on it. It has been running an army of agents around the clock, throwing them at problems from dozens of angles at once, building my software factory on the side, and testing every StateFrame release on Linux before it ships. Since then StateFrame has gone from 0.2.2 to 0.3.40. That's 50 releases in four days.

I know how that sounds. Fifty releases is either a team firing on all cylinders or a project that can't sit still. Honestly it's some of both, and this post is the full accounting: what the foundry has been doing, what all those releases actually did, the mistakes I shipped, and what's still broken.

Releases since the foundry came online

50

0.2.2 on Nov 1 through 0.3.40 on Nov 4; 53 counting 0.1.0 to 0.2.1

Changelog entries in those releases

213

Fixes, features, tests and docs, every one written down

StateFrame runs on the foundry

827

Run folders on the box after four days

Agents working as I write this

17

Load average about 55 on 32 threads

The pace didn't slow down as the week went on. It sped up, then settled.

#We beat that server like it owes us money

Quick refresher: the foundry is a Dell PowerEdge R720XD with two old Xeons (32 threads), 286 GB of memory and a Tesla P40 with no fans of its own. I wrote about standing it up. It was supposed to be a test box. It became the place where everything happens.

The agent army. Most of my work now runs as crews of agents, each in its own lane with its own copy of the code. One lane chases a bug, another builds a feature, another reviews what the first two did. The foundry has hosted 76 of those crew lanes so far. As I write this, 17 agents are working on it at once.

Dozens of angles at a time. When something is broken and I don't know why, I don't send one agent to look. I send a swarm, each one told to attack the problem from a different direction: read the code, replay the failure, check the assumptions, try to break the fix. Most of them come back with nothing. The one that comes back with something is worth all the rest.

The software factory. On the side, the foundry has been building my software factory, a local-first app that hands out bounded work to agents, keeps every contract and run, and only trusts evidence it checked itself. It's still a work in progress, and it'll get its own post when it's ready.

And every release. Every StateFrame release has to pass the full test suite on macOS and on Linux before it ships. The Linux half runs here, alongside everything else.

It did not all go smoothly. On the first afternoon the P40 fell off the bus, as in the card stopped answering like it had been unplugged, because the server sets its fan speed from the sensors it knows about and a third-party GPU isn't one of them. It was cooking itself while the fans idled. A power cap, a watchdog and a fan curve that follows the GPU fixed it, and it hasn't dropped once since. The box has been up for three days straight at a load of about 55, with 115 GB of its memory in use. The room it lives in is warm. I've stopped apologizing for that.

The fun part, and I mean that, is that running a box this hard kept finding bugs my laptop never would.

• Slow sandbox start-up on a newer kernel. The foundry runs a much newer Linux kernel than my laptop's VM, and on it every sandboxed command took about 3 seconds just to start. The cause turned out to be about 425 bind mounts per command, one for every program on the PATH. Now it's 7, and in 0.2.12 a sandboxed command starts in about half a second. The same release cut the time from launching StateFrame to its first model request from about 90 seconds to about 45 on a loaded Linux host.
• Busy machines. A host that's actually working hard exposes timing assumptions. MCP servers failed discovery on a busy Linux box because StateFrame waited only 0.2 seconds for something that takes longer under load (0.3.6). Startup checks gave up on healthy interpreters that were just slow (0.2.12).
• A shared model server. When several agents share one model server, requests have to wait in line. In one contended run on 0.3.27, requests rode out 28 queue waits, the longest 193 seconds, with no errors. On the same setup, the previous release failed all 4 of its requests in about 80 seconds.

None of those show up when one person runs one agent on one quiet laptop. They show up the moment you try to run a fleet, which is the whole point.

#Runs that used to stop now finish

The most important thing these releases did is boring to describe: runs that used to stop partway through now finish.

• The re-reading stall. On some multi-file tasks the model kept re-reading files instead of editing them, until the engine stopped it. In 0.2.2, the task that used to stall was submitted with all 14 tests passing in all 5 live runs. Five runs on one task is a small sample, so read that as encouraging, not proven.
• Mid-size repositories. Repositories with more than about a dozen source files could fail before the first request (fixed in 0.3.1), or later in the run when the last fit before sending dropped a page of source (fixed in 0.3.5 and 0.3.28). On a 22-file repository, 0.3.10 refused mid-run in 3 of 3 live runs. 0.3.28 refused in 0 of 3. There's now a test that replays every refusal we've ever recorded, 34 of them, without a model, and they all have to finish.
• Dead ends. Sometimes a run could reach a state where the only tool left on the menu was "done." A shell command that only created a directory could do it (0.3.23). As of 0.3.31, the list of known dead ends is empty, and a test checks that every refusal names an action the model can actually take next.

#Faster, and cheaper to run

• Prompt cache reuse. 0.3.13 turned on the stable tool head by default, so the model server can reuse the start of each request. Over 30 runs, uncached prompt tokens per run fell 25% and wall time per run fell 15%, with no drop in hidden-test scores. In 0.3.14 the full tool list is there from the first request, and the cached share of the prompt rose from 18-22% to 37-47%.
• Resume. Checkpoints are on by default since 0.3.0 and are taken at every step. Resuming at request 110 of a 120-request run reached the next request in 47 seconds, against 72 the old way. Runs with the shell on can resume from a checkpoint since 0.3.25, and runs with a time limit since 0.3.31. Checkpoints got about 4 times smaller in 0.3.39.
• Disk. One fixed scripted run with its environment went from 35.8 MB to 0.9 MB in 0.3.29. Run environments are now cloned from a shared seed, stateframe gc cleans up old runs, and stateframe rehydrate rebuilds a removed environment when you need it back.

Here's the before and after in one place. Every row is a measurement from the release notes, not an estimate.

#New things it can do

Almost all of these are off by default. Turned off, StateFrame sends exactly the same requests it did before the feature existed, byte for byte, and a test checks that for every one of them. That rule is a big part of why shipping this fast doesn't scare me.

• Bring your own tools and habits: skills and MCP tool servers (0.2.4), lifecycle hooks at six points in a run (0.2.6), your repository's own AGENTS.md (0.2.8), and powerups, which bundle a skill, tools and hooks into one package you can switch on (0.3.26).
• Any test command as the check (0.2.7). npm test, go test, cargo test or a shell script all work. A run with no evidence at all is labeled "unverified: review the diff."
• A different model for each kind of step (0.2.9): one for reading, one for planning, one for editing.
• Reviewable commits (0.3.9): the agent can split its work into commits that come out as a patch series, without touching your repository.
• Big repositories (0.3.7): with more than 200 source files, StateFrame now picks up to 12 files to edit from the task and its checks, instead of asking you to name them.
• A second look before finishing (0.3.12, opt-in): the first time the model says it's done, it gets the task back as a checklist. On one harder task the hidden-test score went from 2 of 9 to 8 of 9 in 2 of 3 runs. On an easy task it roughly doubled the requests for nothing, which is why it's off by default.
• Hosted endpoints (0.3.37): StateFrame now checks what an OpenAI-compatible server actually supports and sends only what it can prove the server accepts. Local llama.cpp servers get the same bytes as before.

#The three security fixes

I'd rather tell you about these myself.

• 0.2.5: a proxy set in a profile, or a macOS system proxy, could receive a run's model requests, including the task and the API key. Model requests never go through a proxy now.
• 0.2.12: on Linux, an MCP server whose launch ran past its timeout could keep running without a supervisor. It was still inside its sandbox, but with no time or memory limit. Affected 0.2.4 through 0.2.10.
• 0.3.31: on macOS, sandbox rules that protect parts of your project didn't work on paths with characters like é, so a sandboxed process could write there. That one had been there since 0.1.0.

If you run StateFrame, update.

#Mistakes I shipped

Fifty releases means some of them were cleaning up after the others.

• HTTPS was broken on Python 3.12 and newer from 0.1.0 until 0.3.30. Every https:// request failed before it even connected. All my testing used local servers over plain HTTP, so I didn't notice. There's now a test that flags the kind of code that caused it.
• 0.2.14 was numbered below 0.3.0 by mistake. It's the same code as 0.3.1. Use 0.3.1.
• 0.3.15 through 0.3.21 broke background jobs on macOS whenever a job's process started a child in its own session. 0.3.22 fixed it, and the tests that had quietly assumed the broken behavior were corrected.
• A few releases only changed tests. Not exciting, but every one made the suite that guards the next release more honest.

#What still breaks

• Go and Node checks on Linux. They fail under StateFrame's check memory limit, because those runtimes reserve much more address space than they use. macOS isn't affected. It's listed in the known limitations, and the fix is in progress.
• Upgrading mid-run. A run started on one release can't be resumed on another. Finish it, or restart it, across an upgrade.
• Small samples. Most live numbers in this post come from a handful of runs on small tasks. I'll keep publishing bigger samples as the foundry runs them.

#Why ship this way?

Every one of these releases was small enough to test properly on its own and ship the same day. The proxy fix didn't wait for hooks. The HTTPS fix didn't wait for the next feature. The cost is a long changelog, and I'll take that every time over a big release that sits on a security fix for a week.

The foundry is what makes that pace possible. A crew of agents can build, review and test a fix in parallel while another crew works on something else entirely, and every release still gets a real Linux run on real hardware under real load. One old server did all of that for four days without a break. The next two from the rack are next in line, and I can't wait to see what we break with three.

If you run StateFrame and get different results than mine, open an issue on GitHub with your setup and what happened. The full release list is on the releases page.