OpenAI Decisions vs. Jev: Useful Without Being Perfect

I don't need every AI judgment to be perfect. I need the answers to be useful and the mistakes to be manageable.

We already make this tradeoff with models like Haiku or Luna. Sometimes speed and cost matter more than getting every answer right, especially when a mistake is reversible and another check follows. Suggesting a file to inspect is different from deciding which files to delete.

I wouldn't dismiss a circular saw because it can cut through a limb. I'd care about its guards, controls, and how I use it. These APIs deserve the same treatment.

IndyDevDan's 10 Levels of Jev For Agentic Engineers got me interested in cheap reads. His approach puts a cheap decision before more expensive work: check whether a file matters before loading it into an agent's context, or choose a model before sending it a task. The agent still reads the file when it needs the code.

OpenAI Decisions and Jev both answer these narrow questions with a yes/no probability, a choice, or a score. I compared those judgments, not complete agent runs.

Takeaways

• Both can be useful without being perfect. In the 84-case test, both got roughly 93% of category answers right. The results didn't establish an overall accuracy winner.
• The differences depended on the task. Jev was more consistent in the small routing test; Decisions did better on some other checks. Those results aren't broad provider advantages.
• The question is part of the tool. Wording and option order changed answers. Define what counts, then test examples that include plausible mistakes—not just easy successes.
• Keep mistakes manageable. Use these judgments to suggest files to inspect or flag unsupported copy. Check the evidence before consequential actions. Even a high-probability answer can be wrong.
• Cost and approval both matter. Jev cost about 54% less for the matched requests. I'll use Decisions at work because it's approved, while continuing to explore Jev personally.

Choosing the next coding step

Finding the implementation is a useful place to start. Given a target behavior and several candidate files, which should the agent read first?

For “remove archived documents,” one candidate contained this:

function docs(){return 'storage.delete(archived.id)'}

That returns a string. It doesn't delete anything. The instructions explicitly excluded strings, comments, and contracts, and the correct choice was none. Both APIs chose this file anyway.

I tested twelve new implementation-file cases, asking each API twice per case. Both chose correctly in sixteen of the twenty-four answers. I supplied the candidates; this wasn't a repository search.

Routing asks a different question: which model does the supplied policy require? For a duplicate charge with an unknown cause, the policy required reasoning. Decisions sometimes chose fast; Jev consistently chose reasoning. I didn't run the selected models, so this says nothing about whether either would solve the problem.

Reviewing copy against a brief

The same approach works for reviewing marketing copy: give the model a brief and product facts, then ask whether the draft agrees with them. That's narrower than asking whether the copy is good.

One brief addressed solo tutors who need lesson notes without internet. The product supported offline notes but not multi-device sync. The draft said:

For solo tutors: take notes offline and automatically sync them across all your devices.

Both providers caught the conflict. They also accepted the supported alternative: “Solo tutors: write and read lesson notes on your device, even without internet.” Both gave the expected category answer in all eighteen runs of each example.

I tested 48 marketing cases with three question wordings and three text formats, running each combination twice. That produced 864 category answers per API—not 864 different pieces of copy. Decisions answered 860 correctly; Jev answered 863. These totals count category choices, not yes/no answers or scores. They measure agreement with supplied facts, not persuasion, conversion, or legal compliance.

{
  "model": "gpt-6-luna",
  "input": "facts:\nOf 50 tested users, 45 finished setup in under five minutes and five took eight minutes.\n\ncopy:\nEvery tested user finished setup in under five minutes.",
  "questions": [
    {
      "name": "category",
      "type": "choice",
      "instructions": "Classify the advertising claim against only the supplied facts using the defined categories.",
      "choices": [
        {
          "value": "supported",
          "description": "The facts establish the entire claim at its stated scope."
        },
        {
          "value": "contradicted",
          "description": "At least one assertion conflicts with an explicit fact; this takes precedence over missing evidence."
        },
        {
          "value": "unestablished",
          "description": "No assertion conflicts with an explicit fact, but at least one assertion or extrapolation lacks evidence."
        }
      ]
    }
  ]
}

{
  "model": "jev-1.13.0",
  "state": "facts:\nOf 50 tested users, 45 finished setup in under five minutes and five took eight minutes.\n\ncopy:\nEvery tested user finished setup in under five minutes.",
  "questions": {
    "category": {
      "type": "choice",
      "instructions": "Classify the advertising claim against only the supplied facts using the defined categories.",
      "criteria": {
        "supported": "The facts establish the entire claim at its stated scope.",
        "contradicted": "At least one assertion conflicts with an explicit fact; this takes precedence over missing evidence.",
        "unestablished": "No assertion conflicts with an explicit fact, but at least one assertion or extrapolation lacks evidence."
      }
    }
  }
}

The wording is part of the test

A useful question needs to say what counts. I tested an HTTP-implementation question on three files, asking each API three times per file. Decisions got three of nine answers right; Jev got all nine.

I then clarified which transport calls counted and excluded protocols, cleanup, and callbacks. Both APIs got all nine right on the same files. This was a retest after seeing the failures, not a new accuracy benchmark.

I tested the revised question on sixteen new public Swift files, asking each API six times per file. Both got all 96 answers right. In a separate cross-language test, both missed a Python requests.get(url, timeout=10) call on all six attempts. I don't know whether the Swift-specific wording caused that miss.

Accuracy on new cases

I also chose 84 new cases after the initial wording experiments and fixed the expected answers before making API calls. I asked each API twice per case, giving 168 category answers each. Decisions got 156 right and Jev got 157—roughly 93% for both. Excluding one plan case with an ambiguous expected answer reverses the lead: Decisions got 156 of 166 right; Jev got 155. That isn't a useful basis for choosing a winner.

With paraphrased instructions, Jev got 159 of 168 category answers right; with the options in reverse order, it got 152. Decisions got 156 right in both versions. Each version used the same 84 cases. These are small, selected samples, not general accuracy ratings.

Confidence cutoffs

A confidence cutoff can defer uncertain answers. I tested a rule that accepted an answer only if its selected category had a probability of at least 0.95. It accepted 102 of 168 Decisions answers and 113 of 168 Jev answers; all were correct in the baseline test. The rule deferred the remaining answers, including some correct ones. But adding unrelated background let a wrong Decisions answer through at a probability of 0.97 on both runs. I haven't established a production-safe threshold.

Where the APIs differed

The fresh category results don't establish an overall accuracy winner. Both were around 93%, and excluding one ambiguously labeled case reversed the tiny lead. They also shared the implementation-file weakness.

There were differences on specific tasks. Jev chose the model required by the routing policy in all 108 answers; Decisions did so in 96. All twelve wrong Decisions answers involved the same case, tested with different wordings and formats. Decisions was slightly ahead on work-boundary and visible-assertion checks. With only six cases per check, these are task-specific findings, not general provider advantages.

Cost and approval

Cheap decisions only make sense if their cost fits the work. Both APIs charge for input tokens only. Here are the base-rate estimates for 168 matched requests, each asking three questions:

Jev's price per input token was 58% lower. For the 168 matched requests, the estimated saving was about 54%, because the APIs reported different token counts. These calculations use reported usage, not invoices. They exclude retries, review, and regional or long-context premiums.

Cost isn't the only factor in my choice. Jev hasn't been approved for use at work; OpenAI Decisions has, so I'll be using Decisions there. I'm using Jev personally to learn it right now.

Jev's terms and privacy policy leave me with more questions about work use. Alongside the cost difference, there may also be a privacy difference, depending on the account settings and agreements in place. These tests measured accuracy and cost—not privacy.

TypeSafe does promise not to train on inputs and publishes a data-processing agreement. OpenAI's Decisions data controls include default abuse-monitoring retention of up to 30 days, with additional controls for eligible customers. Approval at work isn't a claim that Jev has weaker protections or that OpenAI retains nothing.

How I tested

I sent matching evidence, questions, and options to gpt-6-luna through Decisions and jev-1.13.0 through Jev. Evidence was synthetic or public. The fresh set contained 80 synthetic cases and four public Swift files. None of those four files contained a direct HTTP transport call.

I checked category answers against their expected labels, used a 0.5 cutoff for yes/no probabilities, and rounded scores to the nearest level. The responses were independently regraded. I didn't combine the different test sets into one accuracy score. All three evidence formats—plain labels, Markdown fences, and serialized JSON—were sent as text strings, not Jev's native-object state. I didn't test each provider's best-tuned setup or measure agent productivity.

I used Swamp to build and smoke-test my OpenAI decision tools and Jev agent tools extensions; the comparisons called the APIs directly.