Running a Small Decision Model (Jev) in Production

What Jev decides, and what stays with Claude

TypeSafe describes Jev as a model that returns a choice, score or yes/no probability against a supplied question menu. It does not draft prose. Our version is pinned to jev-1.13.0, so a moving model alias cannot silently change a live decision rule.

We use it in three narrow ways:

1. Claim support. Given a claim and a supplied passage, does that passage state the claim, including its scope and numbers? An uncertain answer goes to Sonnet. An “unsourced” flag is a prompt to look again, not authority to delete a fact.
2. Claim against source. A separate check compares a proposed statement with the source text that is meant to support it. The same fallback applies.
3. Head-to-head page comparison. Jev chooses a winner on each line of a fixed reader-quality rubric. Code combines those line choices and sends close results to Sonnet. “Rank” in our logs means this editorial comparison, not a Google search position.

The large model still writes, explains, checks the complete page, reads difficult source context and judges cases whose answer does not fit a small menu. A person reviews a page before publication. Deterministic checks, including exact numbers and dates, belong in ordinary code. The split is a workflow choice, not a claim that a decision model can replace an editor.

That source-first habit is also why our Citation-Grade Content method traces a published claim back to evidence. Jev helps triage that work inside our pipeline; it is not a customer-facing Xtrusio feature or a substitute for source review.

How we tested the boundary before using it

The claim-support calibration began with 300 items labelled by Sonnet: 68 labelled true and 232 labelled false. We tuned on one half and checked the other. With the deployed rule, Jev says yes above 0.8, no below 0.1, and sends the middle to Sonnet. On the held-out half, the saved calibration reports 79% coverage and 99.2% agreement with those labels on the items Jev decided, with zero false yes and one false no.

This is agreement with a teacher model on that dataset, not a measured 99.2% truth rate for future articles. Some negative examples paired claims with an unrelated page, which is easier than distinguishing two near-miss passages from the same source. The data does not establish human-verified accuracy across languages, topics or model versions. We have not completed that harder calibration.

For page comparisons, we tested Jev against 10 Sonnet head-to-head rankings. On the 73 rubric lines where Jev made a choice, it agreed on 64, or 88%; another 12 lines were marked unsure. It chose the same overall winner on 8 of 10 comparisons. The two misses were close Jev results, so the production rule lets Jev decide only when its won-minus-lost margin is at least four lines and it has at most four unsure lines. Other results escalate.

These are small calibration sets. They justify a cautious route, not a general claim that Jev judges content as well as Sonnet. A ranking can look decisive while still be wrong about a source, which is why an independent editorial gate remains in place.

What the production logs actually show

We read the current run files on October 17, 2026. The 58 saved page-comparison files contain 53 Jev decisions and five escalations. One run accounts for 26 Jev decisions out of 28 comparisons. The 58 files record 657,642 Jev input tokens in total. These counts describe routing and input, not publishing success, ranking gains or an independent quality score.

The separate decision logs are larger. Across the current saved files, they contain 3,789 claim-support calls and 3,585 claim-versus-source calls. The first stage fell back 1,806 times; the second fell back 1,126 times. Together, that is 2,932 fallbacks out of 7,374 calls, about 40%. An earlier calibration expectation was 21%. Production passages were harder than that expectation: short passage windows and source-language differences both produced flags that a later gate rejected.

For context, the logs record 1,111 decisions on October 11, 1,542 on October 12, 1,290 on October 13, 1,267 on October 14, and 1,670 on October 15. These are complete-day log-entry counts. One entry is one decision call, not one article, customer or correct answer. Later dates were still in progress when we took this snapshot.

The important operating result is therefore more modest than “the small model replaced the big one.” It took most page-comparison decisions in the saved run files. For claim checks, the fallback was frequent enough that we still need to improve calibration and passage selection before depending on a lower fallback rate.

One decision receipt from October 15 shows what the route records. We have removed the claim text, which could reveal the underlying assignment, but kept the logged fields and values:

The saved receipt records at=2026-10-15T06:49:25.174Z, stage=claim-sourced, route=fallback, result=unsure, confidence=0.29, model=jev-1.13.0 and costUsd=0.000018144.

That receipt says Jev declined to settle a claim. It does not tell us whether the later answer was correct. Keeping those two questions separate is how we avoid counting a successful escalation as a successful source check.

What it cost, and what that number leaves out

The same October 17 log readback sums Jev's recorded cost to $0.067410 for claim-support calls and $0.133400 for claim-versus-source calls. A separate link-selection batch recorded 6,465,995 input tokens and $0.27 of Jev usage. These amounts come from our own run records. TypeSafe quotes $0.042 per million input tokens and approximately 100 ms; that latency is the vendor's figure, not one we measured ourselves.

These are Jev charges only. They exclude Sonnet fallback, human review, retrieval, infrastructure and the cost of a wrong call. We cannot turn them into a defensible total savings percentage. In the comparison calibration, Jev used roughly 9,000 to 19,000 input tokens per two-order ranking, against roughly 90,000 to 100,000 Sonnet tokens for the compared ranking workflow. Token counts are useful context, but unlike-for-unlike prompts and uncounted downstream work prevent a clean cost-equivalence claim.

This is the same reporting discipline we recommend in an AI visibility report: put the denominator and method next to the headline number, then keep the business outcome separate.

Where the small judge was wrong or unsure

Three failure modes changed how we use it.

A high score did not make a link good. In an internal-link review, rejected anchor choices carried Jev scores from 0.61 to 0.81. Some wrapped awkward fragments; others pointed to an off-topic page. We added an anchor-shape check and kept the human review. As with an AI visibility score, the number only makes sense beside the observations behind it.

A short source window could miss the evidence. A gate found claim flags caused by a passage that omitted the relevant line, even when the full source supported the statement. The repair is to reopen the source and extend the evidence window, not to delete a true fact because a classifier said “unsourced.”

Arabic source text produced false alarms. In another gate, Jev's unsure or unsourced flags did not survive checking the underlying Arabic text and its transcription. That is a concrete reason not to carry thresholds from English calibration into another language without a new labelled test.

We also keep answer menus bounded and give a “not stated” or “none” option where a forced choice would invent an answer. A comparison page is capped at 3,500 words before Jev sees it, and numerical or date logic remains in code. None of these safeguards makes the model infallible; each narrows what a mistaken choice can do.

What we have not measured yet

We have not run a repeatability study on the same inputs, measured our own end-to-end Jev latency, or tested other decision models against the same source checks and page-comparison set. We have not priced the cost of errors, and the Jev charge is not the full pipeline cost. We also need a harder claim-support calibration with near-miss negatives and human-checked labels before treating the current agreement rate as a reliability estimate.

That list is not a footnote. It determines the next experiment: hold a realistic source set aside, include same-page wrong-scope passages, repeat calls, measure time and fallback spend, and have people adjudicate the disputed cases. Until then, we can say what the logs show about routing and billed Jev usage. We cannot say that this setup is the cheapest or safest judge for someone else's workload.

For a team considering a similar split, start with one narrow decision whose correct answer can be checked. Keep an explicit unsure route, log the model version and every decision, and review its misses before raising automation. The same principle applies when AI-search monitoring becomes a content action: keep the question, evidence, action and later result connected instead of treating one model score as the outcome. For how we apply that standard to published pages, read our Citation-Grade Content method.