A fast model is not necessarily a useful worker.

I learned that while testing local models on an M5 Max. One 35B coding model reached roughly 106 tokens per second on warm runs and passed five small agent-shaped tasks. That was a real result. It was also not enough information to decide where it belonged in an actual automation system.

The test was intentionally small: counting, JSON output, a bash command, a YAML edit, and tool-call-shaped JSON. It told me that the model could be fast and capable inside a narrow box. It did not tell me how it would handle a long context, a messy working directory, a tool failure, several retries, or an implementation step I could not afford to get wrong.

That is the gap I want my cli-agent model to close.

The benchmark is not the system

The local-model result still changed how I think about this. A 35B MoE model being faster than some smaller models is surprising until the active path matters more than the headline parameter count. On this hardware, the local model stopped feeling like a toy.

But I had already found the limits. A warm micro-benchmark is not a cold start. A clean prompt is not a real agent context. A successful one-shot response is not a completed task.

The context window made that especially obvious. At roughly 8K tokens or below, local models could feel fast and useful. At 16K, responses slowed and quality started to wobble. At 32K, models that looked good on smaller tasks could give wrong answers on graph-walk-style work.

I did not need another public leaderboard to tell me that. I needed to know what happened when a model did my work.

An agent invocation should leave a record

cli-agent is a Swamp model that runs a command-line agent through a provider such as Claude, OpenCode, Amp, or Gemini. It can select a model for an invocation, run a prompt in a working directory, and persist a structured record instead of leaving the result as a terminal scrollback mystery.

The record includes the parts I want to compare later:

  • provider and model;
  • a prompt hash and optional tags for grouping comparable runs;
  • success or failure, exit code, retries, and timeout status;
  • wall-clock duration and output size;
  • input and output token counts when the underlying CLI supplies them;
  • cost when it is available; and
  • a bounded output preview for inspection without treating logs as the database.

That is not a benchmark by itself. It is the accounting layer for one.

The important part is that the measurement lives beside the invocation. A run can be tagged with an experiment name, task class, model variant, or expected capability. Later, I should be able to ask a useful question such as: did the local model complete the same bounded task reliably enough to justify sending it there again?

What not to measure badly

There is an easy trap here. If an invocation emitted 500 output tokens and took 30 seconds, it is tempting to call that 16.7 tokens per second and move on.

But an agent's wall-clock duration includes more than model generation. It can include startup, prompt construction, reading files, tool calls, network time, retries, and waiting for a subprocess. That number may be useful for the person waiting on the result, but it is not automatically a clean model-throughput number.

So the record needs to answer several different questions instead of collapsing them into one impressive-looking score:

  • Did the task succeed? A fast failure is still a failure.
  • How long did the user or workflow wait? Wall-clock time matters even when the model was not generating the entire time.
  • How much did it cost? Local does not become free merely because the API bill is zero.
  • How many retries did it need? A model that gets there only after repeated recovery is consuming capacity somewhere.
  • What kind of task and context did it receive? A tiny JSON task should not be compared to repository-scale reasoning.
  • What did the validator or reviewer say? A syntactically valid result can still be the wrong answer.

The goal is not to produce a single “best model” number. It is to build enough evidence to place each model in the right part of the system.

The first useful experiment

My first benchmark had five tiny tasks. The next test should keep the same spirit—bounded, repeatable, and cheap enough to run more than once—but use work that resembles the actual pipeline.

I want to compare a local model and a cloud model on a small set of task classes such as:

  1. classify an incoming request into a known routing category;
  2. extract structured facts from a fixed document into a validated JSON shape;
  3. propose a constrained edit with an explicit diff or patch; and
  4. summarize a bounded context with facts that can be checked against the source.

Each task needs a pass condition before the models run. “The output looked good” is not a pass condition. A schema check, a fixture-based assertion, a known answer, or a human review rubric is better.

Each run should have the same tags: experiment, task class, provider, model, context-size bucket, and whether the answer passed. Then the cli-agent invocation record can hold the operational measurements while a validator holds the quality result.

I do not need hundreds of runs to learn something. Five to ten comparable runs can reveal a bad fit, a flaky tool path, a context cliff, or a model that is cheaper in dollars but expensive in babysitting.

A decision, not a leaderboard

The decision I expect to test is modest:

  • use a local model for bounded classify, summarize, extract, and plan steps when it stays reliable;
  • keep cloud models for long-context repository reasoning and high-stakes implementation work;
  • keep local contexts short enough that latency and quality remain predictable; and
  • keep collecting the same evidence after the initial experiment, because a good benchmark result can decay when the workflow changes.

That is less exciting than declaring that local has won or that a new model has made everything else obsolete. It is also more useful.

The original local benchmark showed me a possibility. The agent record should show whether that possibility survives contact with actual work.

What this post is and is not

This is an experiment design and an instrumentation approach, not a claim that the production comparison has already been completed. The current cli-agent record is useful because it captures invocation metadata, duration, available token counts, cost, retries, and outcome. It still needs representative runs and explicit validators before it can justify a broad conclusion about local versus cloud agents.

When those runs exist, the useful publication will include the task fixtures, the pass criteria, the aggregate results, the failures, and the engineering decision that followed. A negative result would still count. “This model was fast but did not earn this job” is more useful than a clean benchmark score with no operational consequences.

I wrote about the emotional temptation to turn a benchmark into a forecast in Where Benchmarks Are Prophecies. This is the technical counterpart: keep the receipts, define the work, and let the evidence decide where the agent belongs.