Methods and results: three agent-tool development studies

Prepared 23 October 2026. This appendix supports the accompanying articles on khalymon.dev. It reconstructs development experiments from retained run artifacts and coding-session records. No new model trials were run for this analysis. Private inputs and internal identifiers are omitted; the accompanying data contain anonymous measurements.

These studies answer different questions. Release Manager compares two evidence interfaces on matched historical pull requests. Log Analyzer follows successive revisions on one incident, with a separate, earlier two-incident comparison. Runner checks examine whether the measurement system preserved the intended task and correctly classified execution. Their results are not pooled.

Release Manager: eight matched investigations

The main comparison contains eight valid pull-request pairs: six with failures and two green controls. Both arms used Claude Sonnet 5.5 at high effort, the same read-only GitHub evidence interface and a $4 maximum model-cost estimate per arm. Treatment added Release Manager and indexed retrieval helpers. The baseline had tools, but no unrestricted shell. Model, harness and engine configuration were pinned; before/after snapshots checked stability, and arm order followed a predeclared seeded crossover.

There were 128 failed jobs across the six failure cases; 20 cancelled jobs were tracked separately. Specific explanation coverage credits a correct observed failure mechanism or a verified historical match, including evidence in the structured answer. Additional unsupported claims were reviewed separately. Coverage therefore does not measure the correctness of every sentence.

After one reference correction, the baseline had 68 specific, 5 partial and 55 unknown explanations. Treatment had 120 specific, 8 partial and 0 unknown. Pooled coverage was 68/128 = 53.125%, compared with 120/128 = 93.75%, a 40.625 percentage-point increase. The original baseline of 67 specific explanations was amended after source review; the treatment count did not change. Jobs within a pull request are correlated, so 128 is a coverage denominator, not 128 independent experiments. Both arms made 8/8 supported binary CI readiness decisions. On one case, baseline coverage was higher: 16/17 versus 15/17.

A second model, Codex, re-graded a sample of these Release Manager grades only. The unit was one failed CI job and one arm's final answer, graded specific, partial, or unknown against raw job logs, using the specific-coverage rubric above. Partial and unknown had no written definition; working definitions were written for the check. The sample was 20 items drawn with a fixed seed (20261025) from 255 graded items across the six failing pull requests and both arms, stratified by arm and agent grade, with partial grades oversampled (12 specific, 4 partial, and 4 unknown). One item believed to be the later-corrected grade was excluded. Arm labels were hidden and rows were shuffled, but answer wording can reveal the arm. Codex is a model, and the check was not blind: its session had previously seen some of the original grades, and the sheet showed the original agent-written reference mechanism. It used full local raw logs where the bundle was incomplete. It matched the original grades on 20 of 20 items (Cohen's kappa 1.0; Wilson 95% CI for agreement 83.9%-100%). This is a consistency check between two models, not independent validation, and it does not show that a human would grade the same way. It does not apply to the Log Analyzer or runner judgments.

Estimated model cost totaled $14.1892476 versus $5.8052862, a 59.0867% reduction. The sum of individual trial elapsed times was 1,078.600680959 versus 1,625.000276294 seconds, an increase of 50.6582%. These are eight-run sums per arm, not campaign wall-clock durations or per-case medians. Timing includes model and tool I/O during execution and excludes the subsequent after-snapshot fetch.

No retained trial reached the $4 cap. The highest estimate was $2.847814. Median wall time, the mean of the two central values among the eight wall times, was 147.1 seconds for the GitHub-tools arm and 233.0 seconds with Release Manager. Summed recorded model API time was 800.1 versus 515.9 seconds. Summed recorded tool-call time was 255.0 versus 1,100.4 seconds. Model API time and recorded tool time overlap and are not an additive partition of wall time. The treatment spent much more time in recorded tool calls and less in model API time. Per-trial figures, without repository identifiers, are in release manager trial timing (CSV). Pair numbers there follow retained trial-file order. They are not the case letters in the coverage table.

One additional baseline attempt was interrupted by a GitHub transport failure. Its $1.1026072 estimate was retained separately; no treatment was launched, and the incomplete case was excluded from paired results. There were no paid retries. The paired totals are not total research spending.

Release Manager: separate development checks

An offline replay replaced a 384,702-character initial bundle with a 6,319-character index, reducing upfront text by 98.3574%. All 20/20 job records remained deliverable and engine classifications were unchanged. The replay used zero model calls and zero live calls. This measures initial representation size, not total investigation tokens or answer quality.

A subsequent tool-assisted candidate used ten known development cases, without a fresh baseline arm. Specific explanation coverage rose from a separately corrected 166/194 (85.5670%) to 188/194 (96.9072%), or 11.3402 percentage points. The earlier unamended count was 165. Valid final answers increased from 9/10 to 10/10. The final candidate delivered 221/221 non-green job records, accounted for 862/862 finding IDs, investigated 777 and explicitly left 85 uninvestigated. Four malformed submissions were repaired within their original runs; there were zero paid trial retries. These measures distinguish delivery, accounting, investigation and completion. None certifies all-claim correctness or production reliability.

Log Analyzer: eight successive versions

Each version was run three times on the same development incident, using Sonnet 5.5 at high effort in Claude Code 2.1.285. The analyzer and its skill changed together, and investigation breadth varied. These are sequential development comparisons, not an isolated randomized test of one feature.

The recorded sequence is compact facts; first verification workflow; concise verification output; model-carried final rendering; direct checked-answer delivery; request and cache evidence; derived facts; and selectable claims. All eight stages appear in the accompanying data. Each metric is a separately calculated median over three runs; the medians need not describe one representative run.

The first verification workflow increased median tokens from 850,737 to 1,484,025 and estimated cost from $0.561964 to $0.764481: 74.4399% more tokens and 36.0374% more cost. Focused review still identified manual errors in all three reports.

The final selectable-claims version used median 722,905 tokens, 18 turns, 102.004 agent seconds and $0.45539. Against the immediately preceding derived-facts version—1,322,216 tokens, 22 turns, 136.582 seconds and $0.7480542—the reductions were 45.3263% in tokens, 18.1818% in turns, 25.3167% in agent time and 39.1234% in estimated cost. Against the earlier compact-facts version, the net reductions were smaller: 15.0260%, 14.2857%, 10.3829% and 18.9646%, respectively. The 45% figure compares two specialized versions, not a general-purpose baseline.

All versions retained 3/3 required evidence relationships in each run. That is preserved required-link coverage, not demonstrated improvement in overall accuracy. Token totals include cache reads and cache creation, so they are not unique input size. Agent time excludes build/setup and post-agent verification.

The earlier two-incident comparison

A separate earlier comparison used two distinct incidents, with one run per model, workflow and incident. Its baseline had ordinary shell, file-reading and scripting tools. Values here are sums over two runs, not the later three-run medians. With the specialized analyzer and skill, Sonnet's estimated cost changed from $1.5125614 to $1.674876 (+10.7311%), while agent time fell from 322.729 to 283.837 seconds. For Opus, cost changed from $3.1443544 to $2.7075072 (−13.8931%), and time fell from 474.624 to 400.374 seconds. Each arm retained 6/7 required links. The benefit depended on the model and metric; this small comparison does not establish general superiority.

What verification and runner checks established

In one intermediate Log Analyzer version, all 3/3 runs rewrote a generated checked section. Direct post-agent delivery subsequently preserved checked objects in 3/3 runs, adding a separately measured median 0.395 seconds and no model calls. That timing belongs to this intermediate stage. In the final version, the same pool of 131 candidate claims passed fresh verification for each run; agents selected 38, 36 and 39, and the selected objects matched the delivered checked facts. These are repeated selections from one incident. Manual interpretation remained outside that guarantee, and focused review still found errors in surrounding prose.

An earlier runner check in the exact lnav evaluation image parsed 746/746, 942/942 and 8,018/8,018 entries, totaling 9,706/9,706 (100%). Path replacement removed the route name. A specific rule broken by that change is unverified: the inspected request-start-id rule matches both the original path and the replacement. Semantic equivalence after sanitization was not established. Complete parsing did not establish semantic equivalence.

One completed trial was falsely marked invalid when progress-event metadata was treated as agent delegation. Replaying its existing transcript after fixing the detector made it valid while preserving 27 turns, 1,462,572 tokens and $0.7257176. No new model call was needed. One verified false exclusion supplies no general false-positive rate. Ground-truth isolation and planted-answer probes found in planning records are not presented as completed experiments.

Interpretation limits

All costs are recorded API-equivalent model estimates, not verified charges or engineering costs. Percentage changes use the preceding stated comparator; coverage changes use percentage points. Reviews were focused and unblinded, not independent human adjudication or exhaustive claim-precision audits. The second-model check above covers only the Release Manager specific-coverage grades. Log Analyzer and runner judgments were made by coding agents and were not independently checked by a human or a second reviewer. Cases influenced development. Historical pull requests were read live at recorded heads and latest available attempts, not replayed exactly as of merge. This evidence supports bounded engineering lessons, without held-out validation, significance claims, universal accuracy estimates or measured development ROI.

Download the recorded measurements

The JSON and CSV files contain anonymous aggregate values. They do not contain private logs or an independently reproducible incident corpus.