Both APIs repeatedly picked the wrong file. I wouldn't let either decide where to edit without reading the source.
IndyDevDan's 10 Levels of Jev For Agentic Engineers got me interested in cheap reads: checking whether a file matters before loading it into an agent's context. His examples also cover routing, guardrails, and context management. I compared those judgments, plus marketing copy. I didn't find a universal winner.
Holdout category results
These are the holdout's compact-batch results: correct category choices only, for 12 cases per topic asked twice. Predicates and scores aren't included. Repeated answers aren't independent cases.
| Holdout topic | Decisions | Jev |
|---|---|---|
| Transport, delegation, preparation, or absence | 24/24 | 23/24 |
| Best implementation file | 16/24 | 16/24 |
| Model route: fast, clarify, or reasoning | 24/24 | 24/24 |
| Earliest necessary history turn | 24/24 | 24/24 |
| Plan readiness | 20/24 | 22/24 |
| Current customer objection | 24/24 | 24/24 |
| Offer disclosure | 24/24 | 24/24 |
| Baseline total (84 cases × 2) | 156/168 | 157/168 |
Asking the identical category question alone gave 156/168 for each provider, versus the batched totals above. I saw no obvious batching penalty in this test.
One plan-readiness label was ambiguous: the checklist required both expiration and invalidation without defining the difference. Excluding that case gives 156/166 for Decisions and 155/166 for Jev. I wouldn't pick a provider from that one-answer difference.
Finding the implementation
In an early Swift probe, Decisions called a protocol declaration and a recording-cleanup file HTTP implementations, with probability 1. Jev correctly rejected both.
I replaced the broad question with this:
Does this source define an HTTP request implementation that calls URLSession.data(...), upload(...), download(...), or a similar HTTP transport? Answer no for protocols alone, notifications, database or file cleanup, and UI callbacks.
On the same three control files, Decisions went from 3/9 correct answers to 9/9. Jev passed both wordings. I wrote the fix after seeing the failures. That's a diagnostic result, not held-out accuracy.
That wording worked on sixteen other public Swift files. In a separate set of twenty-four cross-language snippets, both missed a plain Python requests.get(url, timeout=10) call. I don't know whether naming Swift APIs caused that miss.
The separate holdout still exposed trouble choosing the implementation owner. Both preferred return archive.remove(d) over if(d.archived) storage.delete(d.id), although the instructions said to prefer the visible rule owner. Both also selected a file whose deletion code existed only inside a returned string.
For the target “remove archived documents,” these were the three candidates:
// A
function rename(d){return {...d,title:'Archive'}}
// B
function pin(d){return {...d,pinned:true}}
// C
function docs(){return 'storage.delete(archived.id)'}The question was:
Choose the first file to read for the target behavior: prefer a direct implementation owner over a relevant delegate; if equal choose A before B before C. Choose none when neither an owner nor a delegate is present. Strings, comments, and contracts alone are not matches.
The expected answer was none. Both chose C, even with that question alone.
I supplied the candidates; this wasn't a repository search.
Routing and history
In the separate 78-case agent set, Jev got 108/108 model-route category choices right against Decisions' 96/108. Decisions got 100/108 earliest-necessary-history choices right against Jev's 97/108. Each denominator covers six cases with three wordings, three evidence formats, and two repeats.
For a duplicate charge with an unknown cause, the supplied policy required reasoning. Decisions picked fast in twelve of eighteen requests. Jev picked reasoning in all eighteen. I didn't run the selected models, so this says nothing about whether either would solve the problem.
Jev also contradicted itself on a self-contained invitation request: its category said no history was needed, with probability 1, while its predicate returned 0.88 for needing history and its score rounded into the multiple-turn bucket.
In the holdout, a conversion plan explicitly named LedgerRates service, maximum age, stale fallback, rounding, and storage. Decisions chose ready on both compact-batch repeats; Jev asked for clarification on both. An earlier example hadn't clearly named the rate source; I wouldn't count that one as a confirmed model error.
Checking evidence
Some narrow checks worked well. Both required security review for authorization removal despite passing admin-only tests. Both accepted rejection of a free-plan export when the supplied contract required it.
I wouldn't turn that into permission to execute commands. In a separate command-policy probe, Jev correctly classified git reset --hard HEAD as destructive but returned only 0.46 for requiring review. Where policy already determines the answer, I'd derive the review requirement from the category rather than ask another question.
Test-source checks only inspected visible assertions. They didn't establish executed coverage. Credential-like strings were fabricated, not verified credentials.
Marketing copy
The marketing categories were close overall: Decisions got 860/864 choices right; Jev got 863/864. That's 48 cases with three wordings, three evidence formats, and two repeats—not 864 independent copy samples.
Scores exposed different mistakes. Jev's score treated a price concern without explicit refusal or delay as blocking adoption in seventeen of eighteen requests; its category correctly identified price in all eighteen. It also scored an explicitly resolved usability problem as an objection. Decisions handled those score examples better.
Decisions had a different problem. In four of eighteen requests, all using Markdown evidence, it classified “All customers: try it free for 7 days” as accurately qualified despite terms requiring automatic $12/month renewal. Its separate predicate correctly rejected complete disclosure. Jev's category consistently caught the omission.
Both received these facts as text:
Terms: All customers receive a free 7-day trial, then automatically pay $12/month until canceled. No other conditions.
Copy: All customers: try it free for 7 days.
One failed request used this category question in OpenAI's questions array. The companion predicate and score questions aren't shown.
{
"name": "category",
"type": "choice",
"instructions": "Compare copy with terms. All supplied eligibility, spending, timing, code, and renewal conditions are material. Choose the defined category; explicit misstatements take precedence over omissions.",
"choices": [
{
"value": "qualified-accurately",
"description": "All supplied material conditions are explicitly disclosed and no term is misstated."
},
{
"value": "conditions-omitted",
"description": "At least one material condition is absent, but no explicit statement conflicts with the terms. Silence alone is omission, not misstatement."
},
{
"value": "eligibility-misstated",
"description": "An explicit statement conflicts with eligibility or another offer term, whether or not other conditions are omitted."
}
]
}Jev received the same question under questions.category:
{
"type": "choice",
"instructions": "Compare copy with terms. All supplied eligibility, spending, timing, code, and renewal conditions are material. Choose the defined category; explicit misstatements take precedence over omissions.",
"criteria": {
"qualified-accurately": "All supplied material conditions are explicitly disclosed and no term is misstated.",
"conditions-omitted": "At least one material condition is absent, but no explicit statement conflicts with the terms. Silence alone is omission, not misstatement.",
"eligibility-misstated": "An explicit statement conflicts with eligibility or another offer term, whether or not other conditions are omitted."
}
}The expected category was conditions-omitted: the copy left out automatic renewal, its price, and cancellation. These are comparisons with supplied terms, not requests to invent legal requirements.
These were checks against supplied briefs, facts, and terms. I didn't measure conversion or establish legal compliance.
Confidence didn't settle it
Both got the holdout's baseline category right on both repeats for 78 of the 84 cases.
A preset 0.95 category-probability cutoff accepted 102/168 Decisions answers and 113/168 Jev answers, with no accepted errors. It also deferred the rest. Adding unrelated background broke that result for Decisions: correct offer copy became an alleged omission, with probability 0.97 on both repeats.
Jev had no accepted category errors at 0.95 across the five holdout conditions. I still don't have evidence for a safe production threshold. This is a small, selected set. Both APIs repeated wrong answers, and correct categories could sit beside wrong predicates or scores.
Jev cost less
Both charge for input tokens only. At their published base rates, here's the estimated cost of the holdout's 168 compact-batch requests per provider, each asking three questions:
| Provider | Price per 1M input tokens | Recorded input tokens | Estimated API cost |
|---|---|---|---|
| Decisions | $0.10 | 110,038 | $0.0110 |
| Jev | $0.042 | 119,508 | $0.0050 |
Jev's token rate is 58% lower; its estimated cost for these matched requests was about 54% lower. The providers reported different token counts for the same evidence and questions. At those average request sizes, a million requests would cost about $65.50 for Decisions and $29.88 for Jev.
These are base-rate estimates from returned usage, not invoices. They exclude regional or long-context premiums and any review or retry costs.
What I ran
I used Swamp to build and smoke-test my Jev agent tools and OpenAI decision tools extensions. The comparisons here called the APIs directly.
I sent matching evidence strings, instructions, and option descriptions to gpt-6-luna through OpenAI Decisions and jev-1.13.0 through Jev. Separate sets contained 56 code/text, 78 agent, and 48 marketing cases, each with three wordings, three evidence formats, and two repeats. Most requests asked a yes/no predicate, category choice, and score. Older source probes used their own fixtures; I haven't pooled the sets. Their Jev score rubrics omitted nine numeric labels while preserving the descriptions; later sets preserved both.
The 84-case holdout contained 80 synthetic cases and four fresh public Swift files—all negative for direct HTTP transport. Its five conditions were compact batch, identical category question alone, unrelated background, reversed category option order, and paraphrased instructions. Each comparison changed one factor.
Labels were frozen before calls. I graded predicates at 0.5, categories by exact match, and scores by nearest level. Responses were independently regraded.
I checked the requests against TypeSafe's skill, OpenAI's documentation skill, and their live API guides. Both support independent questions on shared evidence; Jev accepts structured score levels. The holdout used one or three questions per request, with at most 4,653 reported input tokens—well below Jev's documented limits. These weren't capacity tests.
Some probes weren't good application designs. Model routes belong in a Choice, not an ordered score. For history, I'd judge each turn's relevance and count in code. The category comparisons above exclude those scores. Irrelevant background was a deliberate stress test; TypeSafe recommends filtering it out. I didn't test its recommended native-object state, only text strings including serialized JSON. This wasn't a comparison of each provider's best-tuned setup.
These comparisons used synthetic or public evidence. I measured neither actual agent productivity nor secure enforcement. I'd try either API for a bounded judgment, check it on representative examples, and keep reading the source before making an edit.