These are engineering notes from eight historical pull requests, one attempt per arm. They are not a benchmark.
The answers were graded by coding agents involved in building the tool. The grading was unblinded. There was one such review per answer.
A second model, Codex, re-graded 20 of these Release Manager items against raw job logs, and it was not blind: its session had previously seen some of the original grades, the sheet showed the original agent-written reference mechanism, and it used full local raw logs where the bundle was incomplete. It matched the original grades on 20 of 20 items (Cohen’s kappa 1.0; Wilson 95% CI for agreement 83.9%-100%). This is a consistency check between two models, not independent validation, and it does not show that a human would grade the same way.
The most useful result came with a tradeoff. Across the eight pull requests, the same model gave specific explanations for 68 of 128 failed jobs using GitHub evidence tools alone. With Release Manager, it explained 120.
Estimated model cost fell from $14.19 to $5.81, a 59.1% reduction. No trial hit the $4 per-arm estimate cap. The highest estimate was $2.847814.
The sum of the trial execution times rose from 18.0 to 27.1 minutes, or 50.7%. That time is the sum of individual trial elapsed times. Median wall time was 147.1 seconds with GitHub tools and 233.0 seconds with Release Manager, each the mean of the two central values among the eight wall times. Summed recorded model API time was 800.1 seconds versus 515.9 seconds. Summed recorded tool-call time was 255.0 seconds versus 1,100.4 seconds. Model API time and recorded tool time overlap and are not an additive partition of wall time. The treatment spent much more time in recorded tool calls and less in model API time. The per-trial figures are in the method notes, not in this article.
Both approaches got all eight binary CI readiness decisions right. Binary readiness alone would have missed most of the difference.
A correct verdict can hide a shallow investigation
A failed CI check often makes the basic decision straightforward: hold the release. The useful work is explaining what failed, separating mechanisms that need different responses and identifying what remains uncertain.
Release Manager was intended to help the model investigate that evidence. The baseline already had read-only GitHub tools. It could retrieve evidence and reason about it. The treatment added the specialized tool.
Six PRs contained failed jobs; two were green controls with no failed jobs. Specific explanation coverage of those failed jobs increased from 53.1% to 93.8%, a gain of 40.6 percentage points.
The tool did not win on every PR. In one case, the baseline explained 16 of 17 failed jobs, while the treatment explained 15. That case belongs beside the aggregate result.
Eight paired PR investigations, one attempt per arm and PR. The 128 failed jobs are the coverage denominator. Jobs within a pull request are related.
Some evidence was present but easy to miss
An earlier investigation produced an evidence bundle containing 384,702 characters. The agent’s observed reads covered its beginning, but two useful error markers were beyond those ranges.
The missing error messages were in the artifact. The interface made them easy to miss.
Other failures were more structural. A single job label could hide multiple failure mechanisms. A generic test summary could obscure the actual error. An excerpt could keep the stack trace while dropping the message that explained it.
The changes were concrete: preserve individual findings, keep the original error text and make each job’s evidence directly retrievable.
An offline replay replaced the initial bundle with a 6,319-character index, 98.4% smaller, while making evidence for all 20 jobs available. The classifications stayed the same. No model call was needed for that replay. The 98.4% figure is the reduction in that initial text, not in total investigation tokens or cost. This index comes from the replay of the pilot artifact, not from one of the 16 paired trials.
{
"decision": "HOLD_FOR_REVIEW",
"jobs_total": 20,
"signal_summary": {
"total": 267,
"success": 233,
"failure": 17,
"cancelled": 3,
"skipped": 14,
"pending": 0,
"missing": 0
},
"jobs": [
{
"job_id": "job-0001",
"status": "failure",
"classification": "REAL_TEST_FAILURE",
"root_cause": "ASSERTION_FAILURE",
"has_excerpt": true
}
]
}Job names and GitHub job identifiers are omitted from this fragment.
An earlier ten-case run delivered every indexed non-green job record and all tracked finding IDs, yet one final answer was an empty placeholder.
What I took from it
A useful release assistant needs to explain enough of the failures to support the next decision. Binary readiness was too coarse to tell me whether this one helped.
For these eight PRs, the specialized evidence tool let the same model cover substantially more of the investigation at lower estimated model cost. It also made the trials longer. That is the result I can defend, and the tradeoff I would carry into the next evaluation.
This is also why the title includes the slower execution. Leaving it out would make the tool look better while making the result less useful.
Sources and testing context
The accompanying measurements and method notes include recorded values, comparison definitions and limitations.
The paired study used Sonnet 5.5 at high effort, equal $4 per-arm estimate caps, a pinned configuration and seeded crossover ordering. One case interrupted by a GitHub transport failure was excluded from the paired analysis; its launched spending remained recorded separately. The figures above are totals for the eight valid pairs, not all development spending. Costs are estimates reported by the model runtime, not observed subscription charges or the cost of building the tool.
The baseline score was amended from 67 to 68 after checking historical test source. Original reports were preserved. Historical heads were inspected through live evidence, rather than a complete replay of their state at merge time.
AI Agents That Matter discusses evaluating agent cost alongside task quality. It supports the evaluation principle; it does not independently validate these local results.