Our AI Agents Outran Our CI. Here's the Box We Built.

One in ten of our Windows CI jobs waited more than two hours for a runner. The runners were busy 38% of the time.

TL;DR

• Self-hosted CI for AI agents needs room for bursts and a clean machine for every job. Ours is one dedicated box, apart from production and from the machines agents edit on. Linux jobs get a fresh VM, Windows jobs a clean snapshot, and memory sets how many run at once.
• In the 30 days before the move, Windows jobs waited 146 minutes at p90 and 622 at p99, while the slots averaged 38% busy. The queue was the constraint, not compute.
• From a bare BIOS screen, 13 repos were running on the box about 17 hours later.
• The rules for what an agent's pull request needs before it merges are the other half.

Why were agents waiting when the runners were mostly idle?

Because the average hid the tail. Half our Windows jobs started within three minutes, so the median looked healthy. One in five waited over an hour.

Bursts, not average load, set the ceiling. In the last 30 days of our telemetry, Windows demand reached 58 jobs queued or running against two slots, and 46 of those came from one repo across four branches. An average-utilisation chart would have told us to buy nothing.

Two more causes hid in the job data. Jobs pinned to one named runner waited an hour or more 41% of the time, against 16% for jobs that could take any Windows runner. In 54% of the minutes jobs were waiting, a slot sat free: work was stuck behind one machine, not short of machines.

And long jobs held slots for hours. One in eight jobs ran 30 minutes or longer, yet used 57% of all runner time.

We are not alone: GitHub reports agent-started pull requests passed 40% in September 2026, and Anthropic saw CI jobs grow 25 times in six months.

Our 92 days (August 1 to November 1, 2026) covered every job in 21 repos: 12,020 jobs and 83,475 runner-minutes, mostly Windows: about 734 runner-minutes a day to Linux's 195. 20.7% failed, and 31% of the failures were not code:

On GitHub Free with a $1 Actions budget, 693 hosted jobs never ran. The cost was blocked work, not a bill.

Why didn't moving tests to the workstation help?

Because it repeated the laptop's mistake: verification sharing a machine with other work. The old CI laptop (6 cores, 12 threads, 32 GB) ran production and three CI runners. To skip its queue, the agents moved the merge gate onto their own workstation, behind a two-slot queue that became our biggest cross-repo bottleneck.

That workstation's inventory tells the story. Headless Chromium crashed about five times a day under parallel tests, often enough that its crash dialogs had been switched off machine-wide. Ports ran out under agent load. Temp files reached 35 GB and agent worktrees 75 GB. Our agent-skills library ran a 543-suite test pass, about 42 minutes, before every push.

The workstation had a high-end CPU and 64 GB. It was not short of CPU. It was short of isolation: one temp folder, one browser cache, one set of ports, one queue for everything.

What did we build instead?

One machine whose only job is verification, so its failure stops CI and never production. It is a Minisforum MS-A2: Ryzen 9 9955HX (16 cores, 32 threads), 96 GB of DDR5 and a 1 TB NVMe. It runs Ubuntu Server on bare metal, Incus to manage virtual machines, and GARM (GitHub Actions Runner Manager) to turn queued jobs into VMs.

Each choice had a real alternative:

That fits our workload; it is not a shopping list. Choosing how much of a job to hand an agent includes choosing where its work gets checked.

What happens to a single CI job?

A Linux job gets a brand-new VM cloned from a versioned image. The VM runs exactly one job, reports to GitHub and is destroyed. It is ready about 20–23 seconds after launch, with browsers, Node versions and database libraries already baked in instead of downloaded by each job.

Windows differs on purpose. Each Windows VM keeps a persistent identity so its licence activation survives; before every job it restores a clean snapshot, takes one job and shuts down. Windows stays only where it is the product. Repos that used it out of habit moved to Linux: Windows VMs need about 50% more memory and a licence each.

Per-job machines let each repo keep its fixed ports and machine-wide locks unchanged. Public repos never use this fleet; GitHub advises self-hosted runners should almost never serve them. And workflows we export into public projects would have waited forever on a private-only runner group. Choose the runner by the visibility of the repo that runs the workflow.

Why did the 96 GB stay in the CI box?

Because RAM decides how many isolated jobs can run at once. The owner offered to move the 96 GB into a laptop and run CI on 32 GB, and left the final call to our AI architect: "You make the final decision." It stayed: with 32 GB only about two jobs fit at once, which would recreate the queue we were leaving.

Memory is allocated statically, so every VM's RAM counts even while its job idles; vCPUs share cheaply (the large lane went from 6 to 12 vCPU at the same 12 GiB). The deciding number was measured: the lowest free memory over 12 hours of real load was 15.4 GiB, and that stopped a third large slot. Lane caps are admission limits, not a promise that every lane peaks at once.

RAM is also the expensive part. On November 2, 2026, the MS-A2 barebone listed at $799 and a 96 GB DDR5 kit at about $1,645: dated list prices, not what we paid, but the memory costs twice the computer.

Shrink the burst before you buy slots. The telemetry said six to eight Windows slots would keep p90 waits near zero. We built three and cut demand instead: cancel superseded runs, keep the PR check short, run full suites after merge, and stop pinning jobs to one named machine. Cancelling superseded runs takes three lines:

concurrency:
  group: gate-${{ github.event.pull_request.number }}
  cancel-in-progress: true

How do jobs share caches without sharing state?

Jobs share read-only inputs and never writable state. Versioned images carry the runtimes, browsers and database libraries, and a management VM runs an npm proxy and a Docker Hub mirror. Each job still gets its own files, browser profile, database and processes. We rejected shared writable caches: one poisoned cache crosses every job's trust boundary.

Pull request jobs get no production or signing credentials. Prometheus, Grafana and Alertmanager watch the box, and alerts open issues in a private operations repo that GitHub emails to its watchers; we have tested an alert opening and auto-closing its issue. Because a host cannot report its own outage, an hourly GitHub-hosted workflow raises an alert if the box goes quiet for 45 minutes.

Which drills found problems a green job would have hidden?

A passing starter job proves only the happy path, so we forced failures on purpose. Restarting the runner manager mid-job left the job green; it reattached in about four seconds. A cancelled job's VM was deleted cleanly. Two other drills, and one slow lane, found real defects that ordinary runs would have misreported.

Running out of memory looked like a cancellation

The drill killed the memory hog, then systemd's default policy stopped the runner service too. GitHub reported "canceled" and skipped even the always() cleanup steps. DefaultOOMPolicy=continue in the image fixed it: the step now fails with exit code 137 and later steps run. Drill how a failure is reported, not just whether the process dies.

Network isolation did not stop impersonation

Reachability checks passed, but a review found a job could claim another VM's network identity. We added IP and MAC filtering to every job VM, and a spoofing probe was then blocked. It was a review finding, not an attack. Test whether jobs can impersonate each other, not only whether they can connect.

Slow Windows startup was a missing file, not a small VM

The base Windows image lacked .NET's precompiled code for PowerShell, so every PowerShell start recompiled it. After the image fix, PowerShell start went from 840 ms to 165 ms, and two slow UI tests from 404 seconds to 67 and from 513 to 87. Those are individual measurements, not a fleet-wide speed-up.

What do people get wrong about clean runners?

They assume every new failure on a clean machine is "environment". Often the clean machine found a real bug. One motion test drove Node to about 8.3 GiB on a 9 GiB VM because pixel batches held native frame buffers; yielding between frames fixed it, and 773 motion tests passed. A bigger VM would have hidden the leak.

• "Environment issue" can be a real bug. An integrity checker rejected a matching checkout because it applied a hard-link check to directories, which Linux counts differently.
• The obvious hunch can be wrong. A portal had 23 failures in about 1,900 tests on Linux. Pinning the timezone fixed none; readiness races and expired fixture dates were the causes.
• Fix, don't weaken. A care-coordination app's browser failures fell from 51 to 9 in one round. Five real product bugs were filed; no assertion was weakened.

What bar do we hold it to?

The same telemetry that found the problem sets the targets. We measure the box against them from GitHub's Actions history:

One box has limits worth knowing before you copy it. It is one host with one NVMe and no ECC memory, so a hardware failure stops CI, though never production, and everything rebuilds from scripts. And its 32 threads are shared by every repo: sharding one very large suite wider takes PR slots from everyone else, so a big suite needs its own speed-up work, not more shards. The companion article covers how we keep the PR check short.

What should you copy first?

Start with measurement, not hardware. Pull a month of job history, compare queue wait with utilisation, and see whether your peaks are one repo's superseded pushes. Then move one representative repo, prove how its failures are reported, and only then expand. These steps, in order:

1. Measure queue wait next to utilisation.
2. Take verification off production and off the machines people and agents edit on.
3. Give every job a clean machine from a baked image.
4. Treat RAM as the admission budget, with a measured floor.
5. Drill failure reporting and impersonation, and route runners by repo visibility.

Frequently Asked Questions

Does every team using AI coding agents need its own CI box?

No. Measure queue wait, burst demand and infrastructure failures before choosing hardware. Our laptop mixed production with CI runners, and the workstation fallback shared test state across agents. A dedicated host removes both. A team with healthy hosted CI may need fewer triggers or a shorter PR check instead of another machine to run.

Is self-hosted CI cheaper than GitHub-hosted runners?

At our volume, an estimate from GitHub's published per-minute prices is about $555 a month on 4-core hosted runners, before allowances. That assumes the same job durations and excludes our electricity, Windows licences and time. Self-hosted minutes are currently free on GitHub. For us the bigger cost was 693 jobs blocked outright by a $1 budget.

Can self-hosted runners run jobs from public repositories?

They can, but GitHub advises they almost never should, because anyone who opens a pull request can run code on your machine. Our runner group admits selected private repositories only. Public repositories stay on GitHub-hosted runners, which are free for public repos, and exported workflows choose their runner by the visibility of the repo running them.

If verification is becoming the slowest part of your team's work, tell us about your work.

About the author: William Rodriguez is the founder of Analytical Ants. He kept the budget, purchase and approval decisions in this rebuild; AI agents did the research, scripting and migrations under them. The build is done for you. The decisions stay yours. More about Analytical Ants.