# Judging every field for a tenth of a cent: TypeSafe's Jev on our extraction benchmark
> We tested TypeSafe's Jev as a per-field judge in an extraction cascade. It caught invented values almost perfectly and recovered 81% of the accuracy gap between a fast/cheap and a slow/expensive extractor while re-running 38% of records.
Canonical page: https://xmemory.ai/jev-per-field-extraction-judge/
At xmemory, our strength is choosing the right model for each step of our memory workflows. We break complex tasks into focused steps and benchmark models to find the best balance of quality, cost, and latency. Our [white paper](https://arxiv.org/abs/2604.27906) applies this approach to extraction, with separate calls for extracting and validating data.
That made TypeSafe's [Jev](https://docs.typesafe.ai/introduction), which answers typed questions with probabilities in about half a second, a compelling candidate for validation. The results are encouraging: as a per-field judge on our extraction benchmark, it recovered **81%** of the accuracy gap between a fast extractor and a slow one while re-running only **38%** of records, reaching **55.33%** output accuracy. Decomposition mattered: asked about a whole record at once, it did no better than a coin flip.
## What Jev is
Jev is TypeSafe's first "System One" model. You send it a *state* (text or JSON) and a map of typed questions, and it returns structured answers:
- **Noul**: a yes/no question, answered with the probability of yes.
- **Choice**: pick one option from a set you define, with a probability for every option.
- **Score**: rate the state against ordered levels.
All questions in a request see the same state and are evaluated independently, in parallel. TypeSafe's docs say extra questions add little latency, and that matched what we saw: our calls carried a median of 136 questions and returned in a median of 0.51 seconds. Pricing is per input token ($0.042 per million for jev-1.13), and output is free. What Jev does not do is write text. It can't produce a corrected value, only judge one. That makes it a candidate judge, not an extractor.
## The benchmark
The dataset is the insurance-claims extraction set from our white paper. It is a cleaned-up version of the insurance subset of [Cleanlab's structured-output benchmark](https://cleanlab.ai/blog/structured-output-benchmark/), published with its schema at [github.com/xmemory-ai/datasets](https://github.com/xmemory-ai/datasets/tree/main/extraction). It has 30 claims written the way claims actually arrive: emails, call-log transcripts, portal forms, in-person intake sheets, one of them with OCR noise. Dates show up as "Jan 20, 2024", "16-8-2025", "yesterday afternoon" and "10-06-23/09-06-26 DD/MM/YY". Amounts show up as "$15.6k".
Each claim maps to a schema with four sections and 20 fields:
- 5 enums: channel, coverage type, object type, incident type, location type
- 4 dates
- 3 integers
- 8 identifiers or names.
Insured objects are a list. Scoring is exact match after light normalisation of case, punctuation and spaces, reported at three levels:
- **Field-level F1.**
- **Object-level accuracy**: every field of an object is right.
- **Output accuracy**: the entire record is exactly right. It is the strictest number, and the one this post focuses on.
In the white paper, the xmemory configuration with a judge in the loop reaches **90.42%** object-level and **62.67%** output accuracy on this benchmark, ahead of every one-shot frontier structured-output baseline we tested; those one-shot baselines land between **34% and 62%** output accuracy. Small per-field errors compound quickly.
## The experiment: a two-speed cascade
We set up the simplest cascade that could benefit from a judge. The extractor is Gemini 3.6 Flash in two configurations:
- **Minimal thinking**: fast and cheap.
- **High thinking**: slower, and about **12×** the output tokens per claim (roughly 3,650 against 310).
Jev looks at every minimal-thinking extraction. Only the records it flags are extracted again with high thinking.
We extracted every claim 10 times in each configuration, with response caching off so that each run is an independent sample: 600 extractions in total.
- Minimal thinking gets **39.67%** of records exactly right.
- High thinking gets **59.00%**.
The question is how much of that 19.33-point gap you can keep while paying for high thinking only when Jev asks for it.
## Designing the judge
TypeSafe's guidance, and their [extraction-cascade cookbook](https://docs.typesafe.ai/cookbooks/sde_cascade) in particular, push hard towards decomposition: many narrow questions instead of one broad one. We followed the cookbook closely.
**The state** is everything a careful human reviewer would look at: the extraction instruction, the JSON schema, the claim text and the extracted record.
**The questions** are generated from the schema for each extraction:
- **Every filled field gets seven yes/no questions**, each phrased so that *yes means wrong*:
- Is the value `hallucinated`, meaning not supported by the text?
- Is it `off_target`, pulled from incidental text?
- Is it `unreasonable`? Literally `"a reasonable person would not have extracted this value"`.
- Is it `incomplete`?
- Is it a `format_violation`?
- Does it mismatch the field's name or description?
- Is it the wrong type?
- **Every empty field gets one question**: does the text contain this information, which would make the empty value wrong?
- **Every enum field also gets a Choice** over the schema's values plus `not_stated`. Jev answers it independently of the extractor. The field's score is one minus the probability Jev gives the extractor's value.
- **Objects and the whole record get their own questions.**
- Each insured object gets "is this object invented or duplicated?".
- The list gets "is an object missing?".
- For comparison only, the whole record gets a single "is this extraction wrong?".
Here is one real question, as sent for a building ID the extractor produced:
```json title="One of 142 questions in a single Jev call (description abridged)"
{
"type": "noul",
"instructions": {
"field_spec": {
"path": "insured_objects[0].object_id",
"type": "string",
"description": "Unique identifier for insured object. … For buildings, use PROP-XXXXXX format. …"
},
"extracted_field": "PROP-990000",
"main_question": "Is the `extracted_field` unsupported by, or absent from, the source text?"
},
"criteria": {
"true": "the `extracted_field` is a hallucination -- not supported by, or absent from, the source text",
"false": "the `extracted_field` is supported by the source text"
}
}
```
That comes to a median of **136 questions per call**: between 68 and 180, depending on how much of the record is filled. All of them go in one request, median latency was **0.51 s** (1.03 s at the 90th percentile).
**Aggregation** is deliberately blunt. A field's score is the highest P(wrong) among its questions, and a record's score is the highest field score. We take the maximum rather than the mean so that one confident red flag isn't averaged away by nineteen healthy fields. A record whose score passes the threshold gets re-extracted.
## What Jev does well
### It finds the wrong fields
Across the 300 minimal-thinking extractions there are 5,874 fields, and 203 of them are wrong (3.5%). The baseline to beat is self-consistency: extract the same claim 10 times and flag fields whose value changes between runs. That is the strongest cheap-ish signal we know of, and it costs nine extra extractions per record.
- **Ranking.** Jev ranks wrong fields above right ones with an AUROC of **0.851**, against **0.758** for self-consistency.
- **Confident flags.** The gap is widest at the top of the ranking, which is where automation happens. Jev's PR-AUC is **0.519** against **0.245**. At a threshold where 9 of 10 flags are real errors, Jev catches **40%** of all errors. Self-consistency never reaches 90% precision at any threshold.
> Image: Dumbbell chart of field-level AUROC for Jev and 10-run self-consistency. All fields: Jev 0.851, self-consistency 0.758. Enum 0.835 vs 0.624, date 0.921 vs 0.760, integer 0.988 vs 0.550, ID or name 0.862 vs 0.870. Extra values 0.998 vs 0.839, wrong values 0.793 vs 0.705, missing values 0.735 vs 0.740.
### It almost never misses an invented value
The standout result is on values the ground truth leaves empty. These are fields where the extractor produced something the text does not support, and they are almost always invented. Jev separates them from correct fields with an AUROC of **0.998**, and flags **84%** of them at 90% precision.
One claim shows what that looks like. Claim CLM-664028 is a portal form about a customer who slipped in the claimant's shop at "9900 Commerce Blvd". The text gives no ID for the building. On one run the extractor returned `PROP-990000`, apparently built from the schema's `PROP-XXXXXX` pattern and the street number. Every other value in that record was right, including the conversion of the DD/MM/YY policy dates. Here are Jev's answers for the invented ID next to a correct one from the same call:
| Question | `object_id` = `PROP-990000` (invented) | `claim_id` = `CLM-664028` (correct) |
|---|---|---|
| `hallucinated` | **0.90** | 0.01 |
| `off_target` | **0.80** | 0.03 |
| `unreasonable` | 0.55 | 0.02 |
| `incomplete` | 0.17 | 0.03 |
| `name_desc_mismatch` | 0.13 | 0.03 |
| `format_violation` | 0.10 | 0.04 |
| `type_mismatch` | 0.06 | 0.06 |
| **Field score (highest)** | **0.90** | **0.06** |
Minimal thinking got that claim exactly right in 5 of its 10 runs. The other five invented an ID, three different ways: `PROP-990000` three times, and `LIAB-664028` and `PROP-664028` once each. Jev scored the five wrong records **0.83–0.90** and the five right ones **0.71–0.74**. At the threshold we ended up using, it re-ran exactly the five wrong records, and high thinking fixed all five.
Self-consistency sees the same instability from the other side. Because the ID varies between runs, every run of this claim looks suspicious to it, the right ones included, and it sends all ten to the slow path.
### Dates and numbers were its best field kinds
TypeSafe's documentation is candid that Jev 1.13 [isn't a calculator and reads dates as text](https://docs.typesafe.ai/model-jaggedness/jev-1.13), so we expected dates and amounts to be its weak spot. They were its strongest:
- **Dates:** AUROC **0.921**, against 0.760 for self-consistency.
- **Integers:** AUROC **0.988**, against 0.550.
The judge never has to do date arithmetic. It only has to notice whether the extracted value is what the text says, and that is a reading task. The integer result rests on only 12 errors, so treat it as encouraging rather than settled.
### The score has a knee you can put a threshold on
Jev's answers are probabilities, but they aren't literal ones. Among fields whose highest P(wrong) is below 0.7, at most 13% are actually wrong. Between 0.7 and 0.8 it is **48%**, and above 0.8 it is **91–100%**.
> Image: Column chart of the share of fields actually wrong by Jev's highest P(wrong): under 13% in every bin below 0.7, 48% for 0.7–0.8, 91% for 0.8–0.9 and 100% for 0.9–1.0, against a perfect-calibration diagonal.
That is exactly the shape you want for a gate: a long quiet floor, then a sharp knee. It also made the threshold easy to choose. We picked it leave-one-claim-out: for each claim, the threshold is tuned on the other 29 claims and then applied to the held-out one. All 30 folds chose the same threshold, **0.78**.
### The cascade works
Put together, the gate does what it is supposed to do.
> Image: Step curves of output accuracy against the share of records re-extracted with high thinking. Jev's per-field signal stays above 10-run self-consistency and well above random escalation, reaching 55.3% at 37.7% re-run; the whole-record Jev question stays below random; minimal thinking alone is 39.7%, all high thinking 59.0%, and an oracle 63.0%.
*Run on 17 September 2026 with jev-1.13.0 and Gemini 3.6 Flash; 30 claims × 10 runs per configuration. Dots mark held-out operating points.*
| Escalation signal | Records re-run | Output accuracy | Random escalation at the same rate |
|---|---|---|---|
| Minimal thinking only | 0% | 39.67% | |
| **Jev, per field** | **37.7%** | **55.33%** | 46.95% |
| Self-consistency, 10 runs | 37.3% | 53.67% | 46.88% |
| Self-consistency, 3 runs | 18.7% | 47.33% | 43.28% |
| Jev, one whole-record question | 39.0% | 45.00% | 47.21% |
| All high thinking | 100% | 59.00% | |
- **Against the extremes.** Re-running the 37.7% of records Jev flags recovers **81%** of the gap between minimal and high thinking, with **62% fewer** high-thinking calls. Object-level accuracy (86.19% vs 87.20%) and field F1 (96.89% vs 96.92%) end up close to the all-high-thinking numbers.
- **Against self-consistency.** Jev leads or ties at every re-run budget. The clearest case is the ≤25% budget: **51.0% against 48.0%**. At the 40% budget the two are within run-to-run noise, and the difference is cost: nine extra extractions per record for self-consistency, one call costing about a tenth of a cent for Jev.
- **Headroom.** An oracle that re-runs exactly the records high thinking fixes reaches 63.0% while re-running 23.3%, so a better judge still has room to win.
### Decomposition is the whole trick
The single whole-record question ("is this extraction wrong?") scored an AUROC of **0.497** at claim level. In the cascade it did worse than random escalation: 45.00% against 47.21% at the same rate. Taking the highest of the per-field scores for the same records gives a claim-level AUROC of **0.703**, and catches 65% of wrong records at 90% precision.
Jev's own documentation lists "hiding several judgments inside one question" among the things to avoid, and this benchmark shows why. The per-field questions also differ among themselves:
- `hallucinated`, `off_target` and `unreasonable` carry most of the confident signal.
- `type_mismatch` is barely better than chance on a schema-constrained extraction, where the types are rarely wrong.
## How we kept ourselves honest
- **Labels agree with the scorer.** Every field was labelled correct, wrong, extra or missing using the benchmark scorer's own matching rules, and our labels agree with the official scorer on all 600 records.
- **Thresholds were chosen on held-out claims.** For each claim, the threshold was picked on the other 29 claims under a 40% re-run cap, then applied to the held-out claim. None of the reported numbers is a best point picked after the fact.
- **The comparisons are paired.** All signals were evaluated offline on the same 600 extractions, so every row in the table above sees identical inputs.
- **Every cascade was re-scored** with our standard benchmark evaluator, and the numbers matched.
- **Jev isn't perfectly deterministic.** Sending identical requests twice moved individual answers by up to 0.10, and a record's highest score by at most 0.02. We pinned `jev-1.13.0` and cached responses so that the analysis is reproducible.
## Where it isn't the right tool
This was an independent evaluation, and Jev has clear edges:
- **Omissions.** It is weak at spotting a field the extractor left empty. The AUROC is 0.735, no better than self-consistency.
- **Plausible but wrong values.** A wrong ID prefix, a date off by one day, Email instead of Phone: Jev ranks these above correct values (AUROC 0.793), but never with enough confidence for 90% precision.
- **Fixes.** It cannot correct what it flags. A generator, or a list of candidates for it to choose from, still has to produce the right value.
## What this means for xmemory
xmemory already has two extraction modes: FAST by default and DEEP for important writes, and the result above maps directly onto them. Extract with FAST, ask a per-field judge, and escalate to DEEP only when it flags a field. The other obvious use is a write-time guard against invented values, the one thing Jev does almost perfectly.
Jev is no longer the only candidate for that job. [AnyJev](https://github.com/nokia-applied-research/AnyJev) from Nokia Applied Research turns an open-weight LLM into a Jev-style decision model, reading answers from the token distribution with no generation and no fine-tuning, and [DiffusionGemma-Jev](https://explainx.ai/blog/diffusiongemma-jev-vllm-open-source-2026) runs Google's DiffusionGemma as a structured decision model through a [merged vLLM change](https://github.com/vllm-project/vllm/pull/57250), filling up to 85 answer slots in a single parallel denoising step; its author reports it roughly level with Jev on early classification tests. Neither is affiliated with TypeSafe, and both point the same way: a judge answering many independent questions has no reason to produce its answers one token at a time, and there is real demand for judges that run on your own hardware.
So the next experiments are the FAST-to-DEEP cascade and the write-time guard inside xmemory's own write path, and this same per-field cascade with the open-weight judges. We will report those numbers.
## References
- [From Unstructured Recall to Schema-Grounded Memory](https://arxiv.org/abs/2604.27906), the xmemory white paper
- [Insurance-claims extraction dataset and schema](https://github.com/xmemory-ai/datasets/tree/main/extraction), derived from [Cleanlab's structured-output benchmark](https://github.com/cleanlab/structured-output-benchmark)
- [TypeSafe documentation](https://docs.typesafe.ai/introduction), including the [extraction-cascade cookbook](https://docs.typesafe.ai/cookbooks/sde_cascade) and the [Jev 1.13 known limitations](https://docs.typesafe.ai/model-jaggedness/jev-1.13)