In one version of my log analyzer, I told the model to copy a checked section unchanged. In all three runs it edited the section, and the agents reported it as unchanged. Then I moved delivery and evidence-building into code. After that, the delivered facts matched what code had checked, and estimated model cost fell 19% against my first compact version. The prose the model wrote around those facts still had citation and interpretation problems.

Read these limits before the numbers.

  • The data is a recorded incident log, with three runs per version.
  • Coding agents that helped build the tool graded the answers. The grading was unblinded, and no human or second reviewer checked it.
  • The analyzer and its skill (the instructions that tell the agent how to use it) changed together between versions, and the breadth of each investigation varied. So I can’t credit any one change with the whole result.
  • All eight versions ran on a later revision of my tool, which isn’t public, so you can’t rerun them yourself.

Before you call a check a guarantee, write down the exact object and property it covers.

The short version

  • The final version cost 19% less than my first compact version.
  • Same model in all eight versions: Sonnet 5.5 at high effort, in Claude Code 2.1.285. One incident, eight versions, three runs per version. Each cost, token and time figure is a median over three runs.
  • My first verification workflow made things worse: 74% more tokens and 36% more cost, and focused review still found errors in all three reports.
  • When I asked the model to copy a checked section unchanged, all three runs edited it. When code delivered the section instead, all three runs kept it intact.
Version (medians)TokensEstimated model costAgent time
1. Compact facts850,737$0.562114 s
2. First verification workflow1,484,025$0.764162 s
7. Derived facts (the version before the final one)1,322,216$0.748137 s
8. Selectable claims (final)722,905$0.455102 s

What the tool does

The analyzer pulls facts out of a log, gives each one an ID, and the model cites those IDs in its report. Finding a single error is easy. The useful part of an investigation is connecting related events and saying what the log can’t show, so the tool has to hand the model those facts and their limits.

The first compact version returned bounded facts. Its medians were about 851,000 tokens, $0.562 in estimated model cost and 114 seconds of agent time. The token totals include repeated input and cache reads across turns. They are not the size of the log.

My first verification workflow was a regression

The next version asked the model to write typed assertions for a verifier. The verifier could check a specific fact against the source records. The agent had to build each assertion correctly and work out which output fields it needed.

Median tokens rose to 1.48 million, a 74% increase. Estimated model cost rose 36%, and agent time rose 42%. Focused review still found manual errors in all three final reports.

I didn’t take that as a reason to drop verification. The check was useful, but its interface made the model do expensive work, and it covered only part of the answer.

The versions after that changed output size, final rendering, evidence delivery and request context. Shorter verifier output brought the median cost down to $0.525. The version that delivered checked facts directly fixed the rewrite described below, but its median cost rose to $0.640. The derived-facts version pushed it to $0.748. The figure shows all eight versions in order.

Development diary of eight recorded iterations of one log-analysis incident: median token use, estimated model cost and agent time. The first verification workflow regressed, and later changes recovered some of the cost.

The model rewrote a section it was told to keep

In one intermediate version, I asked the model to return the checked section unchanged. All three runs kept the meaning of the selected statuses and counts. All three runs also edited the section, and the agents reported it as unchanged. Focused review also found errors in the prose around it.

The numbers survived. The statements of what they cover didn’t always survive. In the rewrite I looked at, the model changed the format and dropped the note that says the citations are samples, not a full count. An instruction to preserve output did not make the delivered text match the checked object.

The next version took delivery away from the model. The model wrote a small file that listed the facts to include. After the agent finished, a separate process recomputed those facts against the frozen input and rendered the checked objects into the report. All three runs delivered the checked objects without changes. That step added a median of 0.395 seconds and made no model calls.

The final version let the tool build the evidence

Delivery was fixed, but the model still had to construct typed evidence objects. The final version moved that work into the tool too. The first bounded call generated claims tied to the input snapshot. The model picked claims by short ID and wrote its own analysis around them. Code then verified the picked claims again and delivered them.

The final version in three steps. First, the tool generates 131 claims from the input, each with a short ID. Second, the model picks 36 to 39 claim IDs per run and writes the analysis. Third, after the agent finishes, code re-verifies the picked claims against the frozen input and renders them into the report with no model calls. In the report, the checked facts are verified and the prose around them is not.

All three runs got the same pool of 131 generated claims. The runs picked 38, 36 and 39 of them, and every picked object matched the delivered checked fact. All three runs kept the three evidence relationships the investigation required. Every earlier version kept them as well, so that number shows nothing was lost. It doesn’t show better accuracy.

Against the version just before it, median tokens fell from 1,322,216 to 722,905 (45%). Estimated cost fell from $0.748 to $0.455 (39%), and agent time fell from 137 to 102 seconds (25%). Against the first compact version, the drop was 15% in tokens and 19% in cost.

What the check did not cover

The model still had real work to do. It had to choose the relevant facts, look into gaps, weigh contrary evidence and say what the logs could not establish. That part still had problems. In the final version, two runs still ran batches of Python commands for other parts of the investigation, and some manual citation and interpretation problems remained. In one run, focused review found no definite factual error, but that review did not check every claim.

So “verified” applied to a specific thing: the selected facts, recomputed against the frozen input and delivered unchanged. It did not apply to the sentences that cite those facts, to an interpretation the model added, or to the choice of what to investigate.

Where I draw the boundary now

I want deterministic code to handle what it can establish exactly. I want the model to do the judgment work.

Code handlesModel handles
Source identityChoosing questions
Event relationshipsInspecting evidence
ArithmeticConsidering alternatives
Record populationsExplaining uncertainty
Preservation of checked outputGoing beyond the prepared facts, marked as unchecked

The model can go beyond the prepared facts when the investigation needs it, as long as it marks those extra claims as unchecked.

I would keep the final interface. It removed a specific piece of model work and kept the targeted facts intact. I would test it on fresh incidents before I call it a general improvement.

One hypothesis goes beyond these numbers. These experiments compare successive versions of one tool on one incident. They don’t compare a generic, configurable setup with one adapted to a domain. In my own use, versions that hard-code the rules of one log family have worked better than configurable profiles. That is a hypothesis, not a result in this article. If it holds, it argues for tools small and readable enough that an agent can adapt them to a specific use case.

If you build this

Letting the tool build the evidence has a cost. The tool has to know the domain well enough to generate the claims up front, and I didn’t measure the development effort. If you build something similar, this is what I’d do:

  1. Let the tool build the evidence objects first, so the model picks claims by ID instead of constructing them.
  2. Check what falls outside the claim pool. Here the tool generated 131 claims, and each run used 36 to 39 of them. Anything outside that pool goes unchecked.
  3. Before you call a check a guarantee, write down the exact object and property it covers.

Sources and testing context

The numbers come from stored October 2026 evaluation outputs. Each metric is its own median across three runs, so the medians need not describe one run. Agent time excludes build and setup, and it excludes the post-agent verification step. Costs are API-equivalent estimates recorded by the model runtime. They are not subscription charges or the cost of building the tool. Agent time, post-agent verification time, estimated model cost and development cost are different quantities.

Anthropic’s tool-design notes give related guidance on focused tools and evaluation. Anthropic’s agent-evaluation guide separates the execution trace from the outcome. The percentages here come from my local experiments only.

Next: I had to debug the benchmark before I could judge the tool (coming soon)

For professional contact, see the About page.