Updated July 27, 2026 with results from the weekend I spent running the factory on real work.
I exhausted a $20 Claude Pro allowance in what looked like roughly two benchmark queries.
That sentence is accurate from the outside and almost useless for explaining what happened.
The two visible CLI-agent invocations were not two model calls. Each invocation expanded into an agent session with many internal turns, repeated tool use, repository discovery, and a growing context that was read again and again. One review used 16 internal turns and roughly 618,000 cache-read tokens. One implementation used 32 turns and about 1.37 million cache reads.
I built a benchmark around the wrong unit of work, then treated the results as if every provider had received the same test.
They had not.
Two queries were not two queries
I was experimenting with a small software factory: give several models the same engineering task, have them review or implement it, and compare the results. Swamp models and workflows launched the work and preserved the outputs so I could inspect what happened afterward.
At the command line, each run looked like one prompt in and one response out. But a CLI agent is not a chat completion. It can inspect files, search a repository, invoke tools, reconsider its approach, read more files, and continue until it decides it is finished. Unless the harness stops it, the number of turns, tool calls, and cache reads is effectively the agent's decision.
Since midnight, seven Sonnet sessions had accumulated 90 internal turns, roughly 333,000 cache-creation tokens, 3.55 million cache reads, and almost 60,000 output tokens. The two largest sessions accounted for 16 and 32 turns.
This does not mean those cache reads map directly to Claude Pro billing. They do not provide a clean conversion into dollars or subscription allowance, and a CLI agent's reported cost is not an authoritative meter for a flat-rate plan. Where I mention cost below, it is an API-equivalent estimate used as a control signal—not a claim about what Anthropic billed me.
The narrower claim is enough: the allowance disappeared much faster than I expected, and the structured usage evidence showed that the sessions were doing far more work than the two visible commands suggested.
I am glad I discovered this on a $20 Claude plan instead of a $200 plan—or, worse, an employer enterprise account. A larger allowance would not have made the experiment better. It would only have let the mistake run longer and made the account boundary more consequential.
The benchmark was already broken
The exhausted allowance was noticeable. The contaminated comparison was the more important failure.
I intended to compare review and implementation quality. Instead, I allowed each CLI agent to discover its own context inside a large repository. That meant each provider got a different test: different files, different instructions, different tool loops, different stopping conditions, and different amounts of irrelevant context.
A model that searched aggressively could appear slow and expensive while finding a better answer. Another could exit quickly with no edit and still look successful because the process returned zero. A third could get trapped reading the same expanding context. Those outcomes say at least as much about the harness as the underlying model.
Exit zero was especially misleading. I saw runs that completed successfully according to the shell but did not make the requested edit. Later, packet certification could persist passed: false while the workflow continued because the certifier process itself exited successfully.
Process success is not task success.
RSpec contributed its own version of this lesson. A malformed or mismatched test selection can report zero examples and still exit successfully. Green text is not evidence if no test actually ran. Certification now checks that the expected suites and a nonzero number of examples executed before accepting the result.
The original rankings were not trustworthy, so I discarded them.
The smaller benchmark told the truth
The corrective move was not a better warning around an open-ended agent. It was changing the unit of work.
I rebuilt the benchmark around exact-context packets. Every model received the same small set of bytes: the task, relevant source, test contract, output contract, and only the skill guidance needed for that task. Discovery outside that packet was not part of the test, so it was not allowed.
The packet hash traveled with the result. The workspace was isolated. Allowed paths, output shape, elapsed time, changed files, and test expectations were explicit. Provider fallback was disabled. If the requested provider was unavailable, the evidence said that provider was unavailable instead of quietly substituting a different model under the wrong label.
The controlled rerun was less flattering and more useful. The strongest model fully passed three of five tiny tasks and still missed two hidden contracts. Another provider passed every visible suite and failed all five tasks once the hidden checks ran. The local Qwen runs timed out even after I reduced the context further. Sonnet produced useful implementation work, then hit the bounded API-equivalent cost ceiling and stopped exactly as designed.
Those are good benchmark results—not because any model won, but because the failures became legible. A timeout is a timeout. A hidden-test failure is a hidden-test failure. Useful work followed by a budget stop is not relabeled as either total success or worthless failure.
Provider and model ceilings now live at the execution boundary. They cap context, turns, tool calls, output, elapsed time, and API-equivalent spend. The exact values vary by task and provider; the important part is that crossing one ends the complete run, returns nonzero, and leaves a reason behind. A warning that permits the agent to continue is not a limit.
What Swamp adds
Swamp matters here because it gives the factory a deterministic control plane rather than another layer of agent improvisation.
A model defines one typed operation: build a packet, invoke a bounded agent, certify an artifact, or run a test contract. A workflow connects those operations as a declarative dependency graph. Each step persists structured data—provider, timing, usage categories, packet identity, output, changed files, test counts, and the domain result—so the next step can consume evidence instead of scraping terminal prose.
That makes fail-closed behavior possible at more than the shell level. The workflow reads the persisted certification result and refuses to advance when the domain result is false, even if the underlying command exited zero. If an implementation produced no allowed edit, the run can record that outcome without pretending it completed the assignment. If a provider times out, a later retry does not rewrite the first attempt's history.
It also separates orchestration from agency. Swamp decides which exact packet exists, which model may receive it, what evidence must come back, and whether the next step is eligible to run. The agent reasons inside that box. It does not get to redefine the box while being measured.
This is useful beyond benchmarks. A software factory needs receipts for every transition: what was requested, what context was supplied, what changed, what tests actually ran, what the run consumed, and why the workflow accepted or rejected it. Otherwise automation just makes ambiguity faster.
Then I ran it for a weekend
A synthetic benchmark can show whether a model survives a small box. I wanted to know whether the box could deliver actual work.
Over the weekend I used the supervised factory on MomentSavor. It recorded 51 work items. Forty-nine reached terminal done, two were still active when I took this snapshot, and none were aborted. Forty-two of the completed runs delivered PWA, data, or export work. Seven improved the factory itself.
Weekend factory outcomes
Total work items 51
Terminal done 49
Active 2
Aborted 0
Completed product/PWA/data/export 42
Completed factory improvements 7Most accepted changes cleared implementation quickly:
Implementation cycles per completed item
1 cycle 36
2 cycles 12
3 cycles 1
Average: 1.29
Median: 1The provenance is the more important number:
Final implementation provenance
Direct model 20
Model-assisted, then bounded attended correction 10
Attended or manual 19Those categories describe the final recorded change, not every model or human contribution in planning, review, testing, or approval.
They also prevent me from telling a better story than the evidence supports. The factory delivered 49 accepted increments. It did not autonomously build 49 things. Twenty final changes came directly from a model. Ten started with useful model work and needed a bounded correction. Nineteen ultimately depended on attended or manual implementation.
That is controlled delivery—not autonomy.
I was still shaping tasks, reviewing evidence, correcting bounded failures, and sometimes writing the change. The factory made that work repeatable and inspectable. It did not remove me from the loop.
The current factory loop
request / Todoist
|
v
bounded intake packet
|
v
plan -> plan review -> approval
|
v
isolated worktree
+ ignored-state snapshot
+ recorded dispatch
|
v
model or attended implementation
|
v
packet certification
|
v
increment review
|
v
deterministic Swamp tests
|
v
final review -> approval
|
v
commit / push
|
v
Todoist evidence
certification, review, or test failure
`-----------------------------> implementationThe loops matter. A model can exit successfully and still produce no useful edit. A reviewer can reject a plausible change. A test can prove that a provider solved the visible shape and missed the actual contract. Those outcomes return to implementation instead of being renamed success.
Provider setup and transport failures are different. An authentication failure or a broken transport did not test implementation ability, so it should not consume an implementation slot. I have been applying that rule manually. Making provider fallback deterministic is still unfinished work.
Amp Code made the weekend possible
I did not sit at a terminal for the entire weekend.
Amp Code was the outer agent environment running this effort. It kept a long-running thread moving on my development machine—inspecting the repository, invoking the Swamp factory, running tests, updating Todoist, committing accepted increments, and continuing to the next bounded task. Much of that work ran while I was doing something else.
The Amp PWA changed how practical that felt. I could check the thread from my phone, see where the work was, answer a question, redirect a bad assumption, attach a screenshot, or tell it to keep going. I had already admitted that I was semi-vibe-coding the original benchmark from my phone. The difference was that, by the end of the weekend, the phone was not hiding an unbounded process from me. It was a useful window into a bounded one.
Long-running and often unattended is not the same as autonomous. Amp still had authority I delegated. I was still supervising, and the factory still stopped at consequential approval gates. But supervision no longer meant sitting at my desk watching terminal output.
There is also an accounting boundary here. Amp was the outer coding and orchestration environment. The CLI-agent subprocess models were workers invoked inside that effort. The factory records which explicit worker planned, reviewed, or implemented a packet. It does not record the model identity behind the outer Amp interaction, so I am not going to invent one.
Without that distinction it is too easy to say “the agent did it” and lose track of whether I mean Amp coordinating the work, Sonnet reviewing a plan, Haiku editing a file, a local model timing out, or me applying the final bounded correction.
The models are starting to show roles
This is not a universal leaderboard. It is a provisional job board from one application and one weekend.
Haiku was a reliable bounded implementer. Small packet, exact paths, concrete acceptance contract—that was a useful role for it.
Sonnet was the strongest routine planner and reviewer I used. It is also the highest Anthropic tier in this factory. There was no Opus quietly rescuing difficult work.
Big Pickle was a useful free remote option when it was available. A recent request failed with a 401, which is evidence of an authentication problem, not evidence that the model cannot do the work.
DeepSeek V4 Flash produced substantive export code. Its roughly 3.75-million-token report was anomalous and mostly cache reads, so I am not treating that as a clean cost comparison.
Local Qwen had a few small wins. Recent runs were less encouraging: about 15 minutes and roughly 109 KB of activity with no diff. I still want to rebenchmark it with the longer timeouts local models needed in the original implementation before deciding whether the problem is capability, tool latency, timeout policy, or all three.
Gemini remains a planned fallback. Headless authentication is unfinished, so it has not had a fair factory comparison yet.
The useful conclusion is not that one model won. Different stages want different behavior, and provider availability is part of the system whether I like it or not.
Skills are context, too
The first benchmark treated repository and skill instructions as free background. They are not. Every skill routed into a session consumes context and can change behavior.
The factory is designed to route only the skills required for the step and to give each stage a context budget. Planning does not need every implementation guide. A narrow test repair does not need the full repository history. Certification should receive the artifact and contract it is certifying—not an invitation to rediscover the project.
This is not an argument for starving agents of useful information. Missing context creates confident mistakes. The point is to make context selection an explicit part of the factory design rather than an accidental side effect of repository access.
In normal development I still want an agent to follow evidence, inspect neighboring code, and revise a theory. For a benchmark, that freedom creates uncontrolled variance unless discovery itself is what I am testing. For factory work, the right amount of agency depends on the stage. Planning can be broader. Implementation should be bounded by the approved plan and paths. Certification should be narrower and skeptical.
I knew better, but not soon enough
The embarrassing part is that I already knew pieces of this lesson: be specific with inputs, review the outputs, and make a plan before asking an agent to do consequential work.
I did not inspect deeply soon enough.
I was semi-vibe-coding from my phone, checking in between other things and steering when something looked wrong. I trusted previous Swamp models too much because they had produced useful structured records before. And I did not think through what the benchmark request actually meant before I sent it.
Swamp gave me better receipts than terminal scrollback would have. I mistook having observability for having control.
The logs could tell me afterward that a visible invocation had become dozens of turns and millions of cache reads. They could not recover the allowance or make an uncontrolled comparison valid. Deterministic records are necessary, but guardrails have to exist before the run starts.
There is a familiar agent lesson here that I keep relearning in more expensive forms: a vague request does not become precise because it enters a typed pipeline. The pipeline can preserve exactly what happened. I still have to define what should happen.
What changes next
The first next step is deterministic fallback. Today I am still choosing the next provider when one never really starts. The factory needs an explicit order, compatible privacy rules, bounded budgets, and provenance that records who actually did the work. No fallback should be silent.
Then I need better autonomy accounting. done tells me that a bounded change passed the delivery system. It does not tell me whether a model implemented it directly, whether I repaired it, how many provider attempts occurred, or whether a setup failure was mislabeled as an implementation attempt. Those should be queryable fields, not a story reconstructed afterward.
Then the factory needs to keep doing application work. I want to finish the remaining PWA and export tasks, use those workloads to expose the next real factory gaps, and perform an adversarial review of the customer-facing PWA commits. A factory that mostly builds itself can become a very elaborate way to avoid shipping the product.
Only then does the rest of the hardening earn its place: stronger isolation, deterministic retries, clearer timeout policy, better provider accounting, and fewer ambiguous outcomes that need human interpretation.
I already knew parts of this lesson: be specific with inputs, review the output, and think through the plan before pressing run. I did not inspect deeply enough soon enough.
The original failure was not exotic. I let an open-ended agent do open-ended agent things, counted the command that started it as one query, and trusted the green exits for too long.
One weekend later, I do not have an autonomous software factory.
I have a supervised one that delivered 49 accepted changes, usually in one implementation cycle, while I could keep an eye on it from my phone.
That is less magical than autonomy—and much more useful than watching the tokens burn.