I exhausted a $20 Claude Pro allowance in what looked like roughly two benchmark queries.
That sentence is accurate from the outside and almost useless for explaining what happened.
The two visible CLI-agent invocations were not two model calls. Each invocation expanded into an agent session with many internal turns, repeated tool use, repository discovery, and a growing context that was read again and again. One review used 16 internal turns and roughly 618,000 cache-read tokens. One implementation used 32 turns and about 1.37 million cache reads.
I built a benchmark around the wrong unit of work, then treated the results as if every provider had received the same test.
They had not.
Two queries were not two queries
I was experimenting with a small software factory: give several models the same engineering task, have them review or implement it, and compare the results. Swamp models and workflows launched the work and preserved the outputs so I could inspect what happened afterward.
At the command line, each run looked like one prompt in and one response out. But a CLI agent is not a chat completion. It can inspect files, search a repository, invoke tools, reconsider its approach, read more files, and continue until it decides it is finished. Unless the harness stops it, the number of turns, tool calls, and cache reads is effectively the agent's decision.
Since midnight, seven Sonnet sessions had accumulated 90 internal turns, roughly 333,000 cache-creation tokens, 3.55 million cache reads, and almost 60,000 output tokens. The two largest sessions accounted for 16 and 32 turns.
This does not mean those cache reads map directly to Claude Pro billing. They do not provide a clean conversion into dollars or subscription allowance, and a CLI agent's reported cost is not an authoritative meter for a flat-rate plan. Where I mention cost below, it is an API-equivalent estimate used as a control signal—not a claim about what Anthropic billed me.
The narrower claim is enough: the allowance disappeared much faster than I expected, and the structured usage evidence showed that the sessions were doing far more work than the two visible commands suggested.
I am glad I discovered this on a $20 Claude plan instead of a $200 plan—or, worse, an employer enterprise account. A larger allowance would not have made the experiment better. It would only have let the mistake run longer and made the account boundary more consequential.
The benchmark was already broken
The exhausted allowance was noticeable. The contaminated comparison was the more important failure.
I intended to compare review and implementation quality. Instead, I allowed each CLI agent to discover its own context inside a large repository. That meant each provider got a different test: different files, different instructions, different tool loops, different stopping conditions, and different amounts of irrelevant context.
A model that searched aggressively could appear slow and expensive while finding a better answer. Another could exit quickly with no edit and still look successful because the process returned zero. A third could get trapped reading the same expanding context. Those outcomes say at least as much about the harness as the underlying model.
Exit zero was especially misleading. I saw runs that completed successfully according to the shell but did not make the requested edit. Later, packet certification could persist passed: false while the workflow continued because the certifier process itself exited successfully.
Process success is not task success.
RSpec contributed its own version of this lesson. A malformed or mismatched test selection can report zero examples and still exit successfully. Green text is not evidence if no test actually ran. Certification now checks that the expected suites and a nonzero number of examples executed before accepting the result.
The original rankings were not trustworthy, so I discarded them.
The smaller benchmark told the truth
The corrective move was not a better warning around an open-ended agent. It was changing the unit of work.
I rebuilt the benchmark around exact-context packets. Every model received the same small set of bytes: the task, relevant source, test contract, output contract, and only the skill guidance needed for that task. Discovery outside that packet was not part of the test, so it was not allowed.
The packet hash traveled with the result. The workspace was isolated. Allowed paths, output shape, elapsed time, changed files, and test expectations were explicit. Provider fallback was disabled. If the requested provider was unavailable, the evidence said that provider was unavailable instead of quietly substituting a different model under the wrong label.
The controlled rerun was less flattering and more useful. The strongest model fully passed three of five tiny tasks and still missed two hidden contracts. Another provider passed every visible suite and failed all five tasks once the hidden checks ran. The local Qwen runs timed out even after I reduced the context further. Sonnet produced useful implementation work, then hit the bounded API-equivalent cost ceiling and stopped exactly as designed.
Those are good benchmark results—not because any model won, but because the failures became legible. A timeout is a timeout. A hidden-test failure is a hidden-test failure. Useful work followed by a budget stop is not relabeled as either total success or worthless failure.
Provider and model ceilings now live at the execution boundary. They cap context, turns, tool calls, output, elapsed time, and API-equivalent spend. The exact values vary by task and provider; the important part is that crossing one ends the complete run, returns nonzero, and leaves a reason behind. A warning that permits the agent to continue is not a limit.
What Swamp adds
Swamp matters here because it gives the factory a deterministic control plane rather than another layer of agent improvisation.
A model defines one typed operation: build a packet, invoke a bounded agent, certify an artifact, or run a test contract. A workflow connects those operations as a declarative dependency graph. Each step persists structured data—provider, timing, usage categories, packet identity, output, changed files, test counts, and the domain result—so the next step can consume evidence instead of scraping terminal prose.
That makes fail-closed behavior possible at more than the shell level. The workflow reads the persisted certification result and refuses to advance when the domain result is false, even if the underlying command exited zero. If an implementation produced no allowed edit, the run can record that outcome without pretending it completed the assignment. If a provider times out, a later retry does not rewrite the first attempt's history.
It also separates orchestration from agency. Swamp decides which exact packet exists, which model may receive it, what evidence must come back, and whether the next step is eligible to run. The agent reasons inside that box. It does not get to redefine the box while being measured.
This is useful beyond benchmarks. A software factory needs receipts for every transition: what was requested, what context was supplied, what changed, what tests actually ran, what the run consumed, and why the workflow accepted or rejected it. Otherwise automation just makes ambiguity faster.
Skills are context, too
The first benchmark treated repository and skill instructions as free background. They are not. Every skill routed into a session consumes context and can change behavior.
The factory now routes only the skills required for the step and gives each stage a context budget. Planning does not need every implementation guide. A narrow test repair does not need the full repository history. Certification should receive the artifact and contract it is certifying—not an invitation to rediscover the project.
This is not an argument for starving agents of useful information. Missing context creates confident mistakes. The point is to make context selection an explicit part of the factory design rather than an accidental side effect of repository access.
In normal development I still want an agent to follow evidence, inspect neighboring code, and revise a theory. For a benchmark, that freedom creates uncontrolled variance unless discovery itself is what I am testing. For factory work, the right amount of agency depends on the stage. Planning can be broader. Implementation should be bounded by the approved plan and paths. Certification should be narrower and skeptical.
I knew better, but not soon enough
The embarrassing part is that I already knew pieces of this lesson: be specific with inputs, review the outputs, and make a plan before asking an agent to do consequential work.
I did not inspect deeply soon enough.
I was semi-vibe-coding from my phone, checking in between other things and steering when something looked wrong. I trusted previous Swamp models too much because they had produced useful structured records before. And I did not think through what the benchmark request actually meant before I sent it.
Swamp gave me better receipts than terminal scrollback would have. I mistook having observability for having control.
The logs could tell me afterward that a visible invocation had become dozens of turns and millions of cache reads. They could not recover the allowance or make an uncontrolled comparison valid. Deterministic records are necessary, but guardrails have to exist before the run starts.
There is a familiar agent lesson here that I keep relearning in more expensive forms: a vague request does not become precise because it enters a typed pipeline. The pipeline can preserve exactly what happened. I still have to define what should happen.
What changes now
I am keeping provider comparisons exact-context, bounded, and cheap enough to repeat. Real factory work will use small application packets rather than a synthetic leaderboard, with model ceilings chosen for the stage and no silent fallback.
I will treat no-edit exits, zero-example test runs, failed certification, timeouts, and budget stops as first-class outcomes. Some may still contain useful work. None should be allowed to masquerade as completion.
I will keep account boundaries explicit before pressing run, especially when personal experimentation could touch a larger plan or someone else's account.
And I will inspect the first strange result instead of assuming the model, wrapper, and workflow mean the same thing by “done.”
The failure was not exotic. I let an open-ended agent do open-ended agent things, counted the command that started it as one query, and trusted the green exits for too long.
Now the factory stops the bot before it can watch all the tokens burn.