A Green Check Can Prove the Wrong Code. Our Rules for Agent PRs.

Our first draft of a verification standard could be fooled. A manually started check could prove the wrong code, and a fast follow-up push could skip the audit.

TL;DR

• CI rules for AI agents have to bind test results, review and approval to the exact commit being merged. We use four GitHub Actions workflows, a short required PR check, review by risk tier and a machine-readable merge record. On GitHub Free, an audit detects rule-breaking after it lands; it cannot prevent it.
• Workstations edit; CI verifies. A new commit voids every earlier approval. A failing main gets fixed before anything else merges in that repo.
• The standard lives in our shared agent-skills library, and each repo adopts it from there. The dedicated CI box it runs on is the other half of the story.

Why does agent-written code need a written contract?

Because parallel authors multiply the ways evidence goes stale. A contract states what counts as proof, ties that proof to one commit and names who may approve. It also gives the next agent a failure record it can read, instead of trusting a predecessor's summary. An agent's confidence never becomes permission to merge.

Under the standard, every test, build and browser run is a GitHub Actions job on a dedicated box, so GitHub records the evidence. Workstations run only format, lint and typecheck, as editing aids. Each repo declares its contract in two files. AGENTS.md gets a Verification section in a fixed shape:

## Verification
- Gate (required on every PR, budget 10 min, ceiling 20): `npm run gate:fast` · lane `aa-linux`
- Full suite (after merge and nightly, budget 60 min, ceiling 90): `npm run gate:slow` · lane `aa-linux-large` · shards 6
- Needs Windows: none | <list and reason>
- Exceptions: none | <what cannot run in CI and why>

REVIEW.md says what a real failure looks like in that repo, which paths are high-risk, which to skip, the cap on nitpicks, and that every finding needs a file and a line.

Which four workflows does every repo get?

A short required check on every pull request, the full suite after every merge, a nightly run, and a manual probe for diagnosis. Each has a different job, and only one of them can authorise a merge. They are rendered from shared templates, with actions pinned to full commit SHAs and permissions defaulting to read-only.

The heart of gate.yml is short:

name: gate
on:
  pull_request:
    types: [opened, synchronize, reopened, ready_for_review]
permissions:
  contents: read
concurrency:
  group: gate-${{ github.event.pull_request.number }}
  cancel-in-progress: true
jobs:
  gate:
    name: gate
    runs-on: aa-linux
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@<full-commit-sha> # v4
        with:
          ref: ${{ github.event.pull_request.merge_commit_sha }}
          persist-credentials: false
      - run: npm run gate:fast

It runs only on pull_request and checks out the merge commit GitHub built for that event: the PR's code combined with main. A check started by hand cannot stand in for it. A gate that outgrows ten minutes becomes shards plus one aggregating job still named gate, each shard with its own database and temp folder. Probe inputs reach the shell only through environment variables.

How does an agent get a pull request merged?

One agent implements a ticket and pushes. CI tests the exact commit. A fresh agent reviews that same commit. The risk tier decides whether a person must approve. Then the orchestrator checks that nothing on main changed under the PR, posts a merge record and merges that exact commit.

The executor's loop is deliberately narrow. It watches checks with gh pr checks <n> --watch --interval 30, never more often, because every agent shares one GitHub API budget. It reads failures with gh run view <run> --log-failed | tail -200, fixes and pushes. Focused reruns go through probe.yml. After three failed rounds it stops and reports.

The reviewer is a fresh, read-only agent with the diff, the ticket's criteria and REVIEW.md. Findings are tagged Important, Nit or Pre-existing, each with a file and a line, and verified before anyone sees them. Review is capped at three rounds; after round two, only the new diff is reviewed. The gate result for the exact head commit is the test evidence, so we retired the agent that re-ran checks on a workstation.

Which changes need a human?

The changed paths and the size of the change set the tier, never the author. Ordinary product work defaults to Medium. Sensitive changes always need a person, however confident the agents are. An agent's approval never counts as a human's, and any approval covers one commit only.

CI workflow files are always High, and that rule caught us: ten early migrations were labelled Low or Medium despite changing workflow files. We corrected the records openly. A tier system needs a check on its own labels.

Independent review earned its place on a payments-heavy web app, where duplicate webhook deliveries could commit conflicting outcomes. The reviewer rejected the first fix: its held database lock could starve the shared production connection pool under load. The second used a short, expiring claim on the payment record, with a test that fails when the protection is removed, and was approved with 243 tests passing.

What can a merge record do on GitHub Free?

It makes every merge checkable after the fact. GitHub Free does not enforce required checks or reviews on private organisation repos, and the owner chose to stay on Free. So the orchestrator applies the rules before merging, and an audit checks every merge afterwards. This detects violations. It does not prevent them.

Immediately before merging, the orchestrator posts one PR comment whose first line is machine-readable:

aa-merge-record v1 head=<sha> review=APPROVE gate=success run=<url> tier=<low|medium|high> human=<none|login> base=<sha> overlap=none

It merges with --match-head-commit <approved-sha>, so a last-second push cannot slip in. Rebase merging is turned off in repo settings, which Free does enforce.

On every push to main, merge-audit walks every first-parent commit in the push (each commit that landed directly on main), not just the newest. Each needs a successful gate from the real pull-request workflow on the PR's final head, finished before the merge, and a trusted merge record written before the merge and never edited. Direct pushes, rebase merges, fork PRs, force-pushes and missing evidence open a merge-policy issue for a person; the audit never reverts anything. The nightly re-audits from the last good nightly, so a skipped push cannot hide.

The audit cannot prove a human truly reviewed the change. A person still has to be the person.

What happens when main turns red or a test flakes?

A failed full suite makes fixing main the next task in that repo, by a fix or a revert, before any new merge. Whether main is red is recomputed from GitHub's Actions history every time; the main-red issue only displays the answer. So a partial re-run, or an old green run, can never close it by accident.

Nothing is retried automatically; that is our rule, not an industry default. A test that passes only on a rerun is flaky: it leaves the gate but still runs nightly, with an owner, an issue and a 14-day expiry CI enforces, and returns after 20 consecutive passes. Infrastructure failures are counted separately, so a red build says which kind of problem you have.

What did the standard get wrong first?

More than we expected, which is why we reviewed the standard itself like code before rolling it out. Independent reviewers went looking for ways the draft could report an unsafe merge as safe, or let a bad one go unnoticed. Among what they found:

• A manually started check could prove the wrong code.
• A fast follow-up push could make the audit skip a commit.
• An old green run could clear a newer red main.
• An issue used as a checkpoint could be edited.
• A partial re-run could close main-red.

Each hole became a rule. The gate runs only on pull requests, on the exact merge commit. Every first-parent commit is audited. Status comes from Actions history and fixed commit SHAs; an issue is never the source of truth.

How do you know the rules are working?

Measure service levels from Actions history and hold them for two consecutive weeks. Ours: queue wait p90 under 2 minutes on Linux and 5 on Windows, PR gate p95 at most 15 minutes (target 10), infrastructure failures under 2% of jobs, and no job able to see another job's files, ports, browser profile, database or processes.

Weekly, per repo, we also track first-pass gate rate, review rounds, flakes, time spent red and revert rate. Reverts are the guardrail: they catch a team that got faster by getting sloppier.

The short gate is what makes the speed possible. Our agent-skills library's PR check ran 544 Linux test suites across three VMs in about 10 minutes; the workstation run it was built to replace took 42. The opposite lesson came from a repo with more than 10,000 unit tests, where about 22 minutes of one run went to loading TypeScript test files. Sharding a suite like that wider takes PR slots from every other repo on a shared box, so a big suite gets its own speed-up work instead.

What did we borrow from Anthropic, and where do we differ?

The review shape is adapted from how the Claude Code team does code review: findings verified before humans see them, tags for real issues versus nits versus pre-existing code, approval set by risk rather than author, CI workflow files always needing a person, and a cap on review rounds. Two lines from it shaped ours:

"Every finding has to be answered."

"After three rounds, the reviewer asks the author to step back and reconsider the approach."

Where we differ:

• Their High tier needs two humans. Ours needs two independent AI reviews plus one human.
• We kept a fast pre-merge gate. At tens of thousands of merges a day, Anthropic wrote that "we've invested in post-merge and pre-deploy automated testing." At our size a short pre-merge gate is cheap, so we run both.
• Failure output fits an agent's context. "The test harness should not print thousands of useless bytes," from Anthropic's C compiler project, is why agents read only the failed step's last 200 lines.
• No retries and the quarantine rules are ours, though counting infrastructure failures separately echoes Anthropic's infrastructure-noise research.

What do people get wrong about agent review?

• "A green check is proof." It proves a job passed, not that it ran the commit you approved. Bind every result to a commit, the way a traceability spine ties evidence to what a change touched.
• "Retry until it's green." A retry hides a flaky test and teaches everyone to ignore red.
• "Approval covers the pull request." It covers one commit. A new push needs a new gate and a new approval.

Frequently Asked Questions

How do you stop AI agents from merging broken code?

Bind every merge to evidence on one commit. The gate must pass on the exact head commit, a fresh reviewer must approve that same commit, and the risk tier's human approval must exist. If main changed under the PR's files, the gate runs again. Then post a merge record and audit every commit that reaches main.

Does the merge audit enforce branch protection on GitHub Free?

No. It checks what landed against trusted evidence recorded before the merge, and opens an issue when a rule was bypassed. It cannot stop an authorised account from merging, and it cannot prove a human truly reviewed a change from a comment alone. Teams that need server-enforced checks on private repos should evaluate GitHub's paid plans.

Should CI retry failed jobs automatically?

We don't. A test that passes only on a rerun is flaky, and an automatic retry hides it. We quarantine flaky tests with an owner, an issue and a 14-day expiry, and require 20 consecutive passes before they return. Infrastructure failures are counted separately and investigated, so a full disk or a lost runner never turns into background noise.

If your agents produce changes faster than your team can trust them, tell us about your work.

About the author: William Rodriguez is the founder of Analytical Ants. He approved this verification standard and keeps the decisions it leaves to a person: risk-tier approvals, merges that need a human, and anything that touches CI. The build is done for you. The decisions stay yours. More about Analytical Ants.