Judging every field for a tenth of a cent: TypeSafe's Jev on our extraction benchmark

At xmemory, our strength is choosing the right model for each step of our memory workflows. We break complex tasks into focused steps and benchmark models to find the best balance of quality, cost, and latency. Our white paper applies this approach to extraction, with separate calls for extracting and validating data.

That made TypeSafe’s Jev, which answers typed questions with probabilities in about half a second, a compelling candidate for validation. The results are encouraging: as a per-field judge on our extraction benchmark, it recovered 81% of the accuracy gap between a fast extractor and a slow one while re-running only 38% of records, reaching 55.33% output accuracy. Decomposition mattered: asked about a whole record at once, it did no better than a coin flip.

What Jev is

Jev is TypeSafe’s first “System One” model. You send it a state (text or JSON) and a map of typed questions, and it returns structured answers:

• Noul: a yes/no question, answered with the probability of yes.
• Choice: pick one option from a set you define, with a probability for every option.
• Score: rate the state against ordered levels.

All questions in a request see the same state and are evaluated independently, in parallel. TypeSafe’s docs say extra questions add little latency, and that matched what we saw: our calls carried a median of 136 questions and returned in a median of 0.51 seconds. Pricing is per input token ($0.042 per million for jev-1.13), and output is free. What Jev does not do is write text. It can’t produce a corrected value, only judge one. That makes it a candidate judge, not an extractor.

The benchmark

The dataset is the insurance-claims extraction set from our white paper. It is a cleaned-up version of the insurance subset of Cleanlab’s structured-output benchmark, published with its schema at github.com/xmemory-ai/datasets. It has 30 claims written the way claims actually arrive: emails, call-log transcripts, portal forms, in-person intake sheets, one of them with OCR noise. Dates show up as “Jan 20, 2024”, “16-8-2025”, “yesterday afternoon” and “10-06-23/09-06-26 DD/MM/YY”. Amounts show up as “$15.6k”.

Each claim maps to a schema with four sections and 20 fields:

• 5 enums: channel, coverage type, object type, incident type, location type
• 4 dates
• 3 integers
• 8 identifiers or names.

Insured objects are a list. Scoring is exact match after light normalisation of case, punctuation and spaces, reported at three levels:

• Field-level F1.
• Object-level accuracy: every field of an object is right.
• Output accuracy: the entire record is exactly right. It is the strictest number, and the one this post focuses on.

In the white paper, the xmemory configuration with a judge in the loop reaches 90.42% object-level and 62.67% output accuracy on this benchmark, ahead of every one-shot frontier structured-output baseline we tested; those one-shot baselines land between 34% and 62% output accuracy. Small per-field errors compound quickly.

The experiment: a two-speed cascade

We set up the simplest cascade that could benefit from a judge. The extractor is Gemini 3.6 Flash in two configurations:

• Minimal thinking: fast and cheap.
• High thinking: slower, and about 12× the output tokens per claim (roughly 3,650 against 310).

Jev looks at every minimal-thinking extraction. Only the records it flags are extracted again with high thinking.

›The cascade

Gemini 3.6 Flash, minimal thinking→Jev checks every field in one call→any field over the threshold?→re-extract with high thinking

We extracted every claim 10 times in each configuration, with response caching off so that each run is an independent sample: 600 extractions in total.

• Minimal thinking gets 39.67% of records exactly right.
• High thinking gets 59.00%.

The question is how much of that 19.33-point gap you can keep while paying for high thinking only when Jev asks for it.

Designing the judge

TypeSafe’s guidance, and their extraction-cascade cookbook in particular, push hard towards decomposition: many narrow questions instead of one broad one. We followed the cookbook closely.

The state is everything a careful human reviewer would look at: the extraction instruction, the JSON schema, the claim text and the extracted record.

The questions are generated from the schema for each extraction:

• Every filled field gets seven yes/no questions, each phrased so that yes means wrong:
  • Is the value hallucinated, meaning not supported by the text?
  • Is it off_target, pulled from incidental text?
  • Is it unreasonable? Literally "a reasonable person would not have extracted this value".
  • Is it incomplete?
  • Is it a format_violation?
  • Does it mismatch the field’s name or description?
  • Is it the wrong type?
• Every empty field gets one question: does the text contain this information, which would make the empty value wrong?
• Every enum field also gets a Choice over the schema’s values plus not_stated. Jev answers it independently of the extractor. The field’s score is one minus the probability Jev gives the extractor’s value.
• Objects and the whole record get their own questions.
  • Each insured object gets “is this object invented or duplicated?”.
  • The list gets “is an object missing?”.
  • For comparison only, the whole record gets a single “is this extraction wrong?”.

Here is one real question, as sent for a building ID the extractor produced:

One of 142 questions in a single Jev call (description abridged)

{
  "type": "noul",
  "instructions": {
    "field_spec": {
      "path": "insured_objects[0].object_id",
      "type": "string",
      "description": "Unique identifier for insured object. … For buildings, use PROP-XXXXXX format. …"
    },
    "extracted_field": "PROP-990000",
    "main_question": "Is the `extracted_field` unsupported by, or absent from, the source text?"
  },
  "criteria": {
    "true": "the `extracted_field` is a hallucination -- not supported by, or absent from, the source text",
    "false": "the `extracted_field` is supported by the source text"
  }
}

That comes to a median of 136 questions per call: between 68 and 180, depending on how much of the record is filled. All of them go in one request, median latency was 0.51 s (1.03 s at the 90th percentile).

Aggregation is deliberately blunt. A field’s score is the highest P(wrong) among its questions, and a record’s score is the highest field score. We take the maximum rather than the mean so that one confident red flag isn’t averaged away by nineteen healthy fields. A record whose score passes the threshold gets re-extracted.

What Jev does well

It finds the wrong fields

Across the 300 minimal-thinking extractions there are 5,874 fields, and 203 of them are wrong (3.5%). The baseline to beat is self-consistency: extract the same claim 10 times and flag fields whose value changes between runs. That is the strongest cheap-ish signal we know of, and it costs nine extra extractions per record.

• Ranking. Jev ranks wrong fields above right ones with an AUROC of 0.851, against 0.758 for self-consistency.
• Confident flags. The gap is widest at the top of the ranking, which is where automation happens. Jev’s PR-AUC is 0.519 against 0.245. At a threshold where 9 of 10 flags are real errors, Jev catches 40% of all errors. Self-consistency never reaches 90% precision at any threshold.

It almost never misses an invented value

The standout result is on values the ground truth leaves empty. These are fields where the extractor produced something the text does not support, and they are almost always invented. Jev separates them from correct fields with an AUROC of 0.998, and flags 84% of them at 90% precision.

One claim shows what that looks like. Claim CLM-664028 is a portal form about a customer who slipped in the claimant’s shop at “9900 Commerce Blvd”. The text gives no ID for the building. On one run the extractor returned PROP-990000, apparently built from the schema’s PROP-XXXXXX pattern and the street number. Every other value in that record was right, including the conversion of the DD/MM/YY policy dates. Here are Jev’s answers for the invented ID next to a correct one from the same call:

Question	object_id = PROP-990000 (invented)	claim_id = CLM-664028 (correct)
hallucinated	0.90	0.01
off_target	0.80	0.03
unreasonable	0.55	0.02
incomplete	0.17	0.03
name_desc_mismatch	0.13	0.03
format_violation	0.10	0.04
type_mismatch	0.06	0.06
Field score (highest)	0.90	0.06

Minimal thinking got that claim exactly right in 5 of its 10 runs. The other five invented an ID, three different ways: PROP-990000 three times, and LIAB-664028 and PROP-664028 once each. Jev scored the five wrong records 0.83–0.90 and the five right ones 0.71–0.74. At the threshold we ended up using, it re-ran exactly the five wrong records, and high thinking fixed all five.

Self-consistency sees the same instability from the other side. Because the ID varies between runs, every run of this claim looks suspicious to it, the right ones included, and it sends all ten to the slow path.

Dates and numbers were its best field kinds

TypeSafe’s documentation is candid that Jev 1.13 isn’t a calculator and reads dates as text, so we expected dates and amounts to be its weak spot. They were its strongest:

• Dates: AUROC 0.921, against 0.760 for self-consistency.
• Integers: AUROC 0.988, against 0.550.

The judge never has to do date arithmetic. It only has to notice whether the extracted value is what the text says, and that is a reading task. The integer result rests on only 12 errors, so treat it as encouraging rather than settled.

The score has a knee you can put a threshold on

Jev’s answers are probabilities, but they aren’t literal ones. Among fields whose highest P(wrong) is below 0.7, at most 13% are actually wrong. Between 0.7 and 0.8 it is 48%, and above 0.8 it is 91–100%.

That is exactly the shape you want for a gate: a long quiet floor, then a sharp knee. It also made the threshold easy to choose. We picked it leave-one-claim-out: for each claim, the threshold is tuned on the other 29 claims and then applied to the held-out one. All 30 folds chose the same threshold, 0.78.

The cascade works

Put together, the gate does what it is supposed to do.

Run on 17 September 2026 with jev-1.13.0 and Gemini 3.6 Flash; 30 claims × 10 runs per configuration. Dots mark held-out operating points.

Escalation signal	Records re-run	Output accuracy	Random escalation at the same rate
Minimal thinking only	0%	39.67%
Jev, per field	37.7%	55.33%	46.95%
Self-consistency, 10 runs	37.3%	53.67%	46.88%
Self-consistency, 3 runs	18.7%	47.33%	43.28%
Jev, one whole-record question	39.0%	45.00%	47.21%
All high thinking	100%	59.00%

• Against the extremes. Re-running the 37.7% of records Jev flags recovers 81% of the gap between minimal and high thinking, with 62% fewer high-thinking calls. Object-level accuracy (86.19% vs 87.20%) and field F1 (96.89% vs 96.92%) end up close to the all-high-thinking numbers.
• Against self-consistency. Jev leads or ties at every re-run budget. The clearest case is the ≤25% budget: 51.0% against 48.0%. At the 40% budget the two are within run-to-run noise, and the difference is cost: nine extra extractions per record for self-consistency, one call costing about a tenth of a cent for Jev.
• Headroom. An oracle that re-runs exactly the records high thinking fixes reaches 63.0% while re-running 23.3%, so a better judge still has room to win.

Decomposition is the whole trick

The judge design lesson

The same model that ranks wrong fields at 0.85 AUROC ranks wrong records at 0.50 when you ask about the whole record at once.

The single whole-record question (“is this extraction wrong?”) scored an AUROC of 0.497 at claim level. In the cascade it did worse than random escalation: 45.00% against 47.21% at the same rate. Taking the highest of the per-field scores for the same records gives a claim-level AUROC of 0.703, and catches 65% of wrong records at 90% precision.

Jev’s own documentation lists “hiding several judgments inside one question” among the things to avoid, and this benchmark shows why. The per-field questions also differ among themselves:

• hallucinated, off_target and unreasonable carry most of the confident signal.
• type_mismatch is barely better than chance on a schema-constrained extraction, where the types are rarely wrong.

How we kept ourselves honest

• Labels agree with the scorer. Every field was labelled correct, wrong, extra or missing using the benchmark scorer’s own matching rules, and our labels agree with the official scorer on all 600 records.
• Thresholds were chosen on held-out claims. For each claim, the threshold was picked on the other 29 claims under a 40% re-run cap, then applied to the held-out claim. None of the reported numbers is a best point picked after the fact.
• The comparisons are paired. All signals were evaluated offline on the same 600 extractions, so every row in the table above sees identical inputs.
• Every cascade was re-scored with our standard benchmark evaluator, and the numbers matched.
• Jev isn’t perfectly deterministic. Sending identical requests twice moved individual answers by up to 0.10, and a record’s highest score by at most 0.02. We pinned jev-1.13.0 and cached responses so that the analysis is reproducible.

Where it isn’t the right tool

This was an independent evaluation, and Jev has clear edges:

• Omissions. It is weak at spotting a field the extractor left empty. The AUROC is 0.735, no better than self-consistency.
• Plausible but wrong values. A wrong ID prefix, a date off by one day, Email instead of Phone: Jev ranks these above correct values (AUROC 0.793), but never with enough confidence for 90% precision.
• Fixes. It cannot correct what it flags. A generator, or a list of candidates for it to choose from, still has to produce the right value.

What this means for xmemory

xmemory already has two extraction modes: FAST by default and DEEP for important writes, and the result above maps directly onto them. Extract with FAST, ask a per-field judge, and escalate to DEEP only when it flags a field. The other obvious use is a write-time guard against invented values, the one thing Jev does almost perfectly.

Jev is no longer the only candidate for that job. AnyJev from Nokia Applied Research turns an open-weight LLM into a Jev-style decision model, reading answers from the token distribution with no generation and no fine-tuning, and DiffusionGemma-Jev runs Google’s DiffusionGemma as a structured decision model through a merged vLLM change, filling up to 85 answer slots in a single parallel denoising step; its author reports it roughly level with Jev on early classification tests. Neither is affiliated with TypeSafe, and both point the same way: a judge answering many independent questions has no reason to produce its answers one token at a time, and there is real demand for judges that run on your own hardware.

So the next experiments are the FAST-to-DEEP cascade and the write-time guard inside xmemory’s own write path, and this same per-field cascade with the open-weight judges. We will report those numbers.

References

• From Unstructured Recall to Schema-Grounded Memory, the xmemory white paper
• Insurance-claims extraction dataset and schema, derived from Cleanlab’s structured-output benchmark
• TypeSafe documentation, including the extraction-cascade cookbook and the Jev 1.13 known limitations