The fleet grows: meet Foundry-2, the R630 that took over the release gates

When I stood up Foundry-1, I said its Xeons had so many cores I'd stopped counting them.

I have started counting them. There are 16.

Two sockets, eight cores each, 32 threads, from Intel's 2012 generation. At the time that felt like a lot. Then I gave it an agent army, a software factory and every StateFrame release gate to run at the same time, and it found out what "a lot" means in 2026.

#The load average that ate my server

Load average is roughly how many things are waiting their turn, for a CPU or for the disk. On a machine with 32 threads, a load around 32 means it's busy but keeping up.

For the first few days of November, Foundry-1 was barely breaking a sweat. Then on November 10 I scaled the agent crews up, and this happened:

From a peak of 12 to a peak of 234 in one day, and 260 the day after. For hours on end it sat somewhere between 150 and 190. That's five or six things waiting for every thread it has. Picture a coffee shop with 32 baristas and 180 people in line. Nobody is getting coffee quickly, and somebody is going to start yelling.

The crews were the big new load. But every StateFrame release also has to pass the full test suite on Linux before it ships, and at the pace StateFrame has been shipping (111 releases in eleven days), those gates were landing on a box that was already drowning. Gates got slow. Agents got slow. Tests with a timer in them started failing because the machine was too busy to answer in time. One release stopped because a test server didn't start within its 10-second window, at a load of 147. That's a terrible reason for a test to fail, and an even worse reason to hold a release.

The 2012 server was doing its best. Its best was not enough.

#Meet Foundry-2

Second one out of the rack is the Dell PowerEdge R630 I promised in that first post. Ubuntu went on it on November 10, the same day the load went through the roof, and it gated its first StateFrame release about an hour after coming online.

CPU cores

32

Two Xeon E5-2698 v3, 64 threads, from 2014

System memory

128 GB

DDR4-2133, practically futuristic in this basement

Usable disk

~3.6 TB

Spinning drives behind a PERC H730 RAID controller

Rack space

1U

Half the height of Foundry-1

Here's how the two compare:

Quick correction while we're here. Last time I said Foundry-1 had "20+ TB" of storage. I was being generous. It's 16 drives in RAID 60 plus a spare, which comes out to about 11 TB you can actually use. Still a lot of spinning rust, just less than I bragged about.

Foundry-1 still wins on memory by a mile. Foundry-2 wins where it hurt: twice the cores, two generations newer, on faster memory. Test suites are mostly about cores, and that's exactly what the gates needed.

#It did not go quietly

I'd love to tell you I slid it into the rack and it just worked. It did not.

The first night, Foundry-2 hung so hard it stopped answering on the network, and I had to power-cycle it by hand. Every job on it was lost. It turned out memory pressure had it thrashing so badly that it never even got around to killing anything. It just froze. Now every heavy job on it runs under a hard memory limit.

The next morning it went down again.

Then I went and read the server's own event log, which is the part I should have done first. It said the crashes were CPU machine checks, the kind of error a CPU throws when the hardware itself is unhappy, and they started right after the "chassis intrusion" alarm. In other words: right after I'd had the lid off. The prime suspects are something I didn't seat properly, like a memory stick, a riser or the GPU I'd just put in. There were also a couple of "AC power lost" entries, which is server for "someone tripped a breaker."

So yes, it's had more reboots since November 10 than I'd like to admit. It's been up and steady since the last round of fixes yesterday, and through all of it, it's been the box running the release gates.

#Who does what now

The split is simple.

• Foundry-2 runs the release gates. Every StateFrame release's Linux test run lands here now, along with other test-heavy work. On its busiest day it ran over a hundred gate runs. It has the cores to run the suite in parallel without fighting anyone for them.
• Foundry-1 runs the crews. All that memory is room for lots of agents side by side, each with its own copy of the code and its own sandbox. That's what it's actually good at.

Don't get me wrong, neither box is relaxing. On a normal working day both of them still sit at loads of 60 to 90. But that's busy, not drowning, and the difference is that things keep moving now. Gates finish, crews keep working, and tests rarely fail anymore just because the machine was too busy to answer. Moving the gates wasn't the only fix, to be fair. This morning Foundry-1 also got a cheap SSD as scratch space for test runs, because a lot of that load was disk, not CPU. On the small random reads test suites love, the SSD does 47,000 operations a second. The spinning array does about 2,000. That's 23 times faster for the price of a nice lunch.

The models themselves mostly run on rented GPUs, by the way, because rented GPUs are elastic and cheap. When I'm testing a hosted model like DeepSeek Flash, that part goes to the provider. The servers run everything around the models: copies of the code, builds, tests and gates. The P40 does take shifts serving a local model for StateFrame testing.

#About that GPU

The R630 was supposed to get a pair of Tesla P4s, the P40's little 8 GB siblings. One of them did go in, and between crashes we got to play with it. On a card from 2016 with 8 GB of memory, it ran Qwen3.6-35B-A3B, the same open model I test StateFrame with, with most of the model sitting in system memory. With four requests at once it put out about 40 tokens a second between them. Not fast. Not bad at all for a 75-watt card that fits in a 1U server.

Right now it's sitting on my desk. After the crashes I pulled it out, figuring a freshly installed GPU was the most likely suspect. Now I'm not so sure it was guilty, so it may well go back in. Carefully.

And the Hyve server from the same rack is still waiting for its turn. Two servers down, one to go.

#The fleet so far

Two servers from my basement, one from 2012 and one from 2014, together:

Servers

2

With a third waiting in the rack

CPU cores

48

96 threads between them

System memory

~414 GB

286 GB DDR3 plus 128 GB DDR4

Servers rented

0

Both were already in the basement. The models mostly run on rented GPUs

Still not a GPU cluster. Still not going to be mistaken for a datacenter, although my wife's ears might disagree. But a release gate no longer has to wait behind seventeen agent crews to find out whether the code works, and that alone was worth dusting off a second server.

The electric bill has opinions about this. I'm not reading them. Besides, I looked it up:

I'm pretty sure the electric bill still comes out ahead.

If you're running your own fleet of old servers for agent work, I want to hear what your bottleneck turned out to be. Mine was never the GPU. It was cores, then memory, then the disk. Open an issue on GitHub and tell me about yours.