The legal specialist joins the uniform board

klusai/kp-deid-xlmr-560m-legal now appears on the same board as the other detectors, following europriv-bench #47. Its strongest result is still on TAB, the external English court-text benchmark:

That is the best TAB direct-identifier leak rate on the board. The comparison covers 127 documents and 264 DIRECT subjects, using the legal seed0 checkpoint and one Presidio run. The CIs are disjoint on this held-out config. The source ledger also records three legal training seeds, but the interval above belongs to seed0; it is not an interval over training seeds.

The same model leaks 100% of CNPs on ro-synthetic-v1, with a 95% Wilson CI of [99.62–100.00]%: all 1,017 subjects across 1,500 documents, one checkpoint. The source marks that config dev and in_distribution; it is not a held-out win. We report the failure alongside the legal result. This is a specialist, not a general privacy model. The leaderboard exposes both results and their config status.

There is a metric boundary too. TAB’s historical DIRECT score treats an occurrence as covered when any predicted token is redacted, then requires coverage of every occurrence of the entity. Part of a name can survive. This is a direct-identifier leakage proxy, not measured attacker re-identification probability or proof of complete residual removal.

A first CPU measurement, with its limits

The CPU study merged in klusai-models #27 measured the adapters on the same 200 chat-length inputs on an Apple M3 Ultra:

One seed0 checkpoint per model, one timed pass, CPU fp32, batch size one. The intervals come from 5,000 input-bootstrap resamples (RNG seed 120), conditional on this corpus and pass. Other work was running on the shared host. Tokenization, model inference and span alignment are included; model loading, transport, queues and redaction operators are not. These timings do not establish idle-host capacity or end-to-end gateway latency.

What happens next

The MoE fine-tune GPU wave, gateway-traffic benchmark and panel validation are in progress. Human-gated programme steps are being replaced by pre-registered LLM-panel validation. Passing work will be labelled panel-validated; that describes panel review and must never be presented as native-speaker validation. Existing development configs remain development configs until their protocol passes.

The acceptance lines are lower leakage than the pinned Presidio-based deployment baseline with disjoint CIs, preserved utility, and measured CPU latency under the agreed load. Stock Presidio on TAB is not that deployment baseline. We have not yet verified those gateway acceptance lines, a general-model claim, or completed panel validation. The roadmap now separates merged evidence from work in progress and what still has to pass.

What we measured

gateway-traffic-v1 contains synthetic conversations in Romanian, English, German, French, Spanish, Italian, Dutch and Polish. Each language’s test split has 1,000 conversations and 4,506 identifier subjects: 4,266 DIRECT and 240 QUASI. An identifier subject can have repeated mentions across turns; it is not a unique person. The UNSPECIFIED sensitivity slice has no support and is unavailable, not a measured zero. The eight language configurations remain dev. Independent annotation review and semantic subject/source/scenario-family/translation disjointness checks are unfinished. Translations can share families across languages, so we report every language separately. Evaluation protocol

Residual leak measures the fraction of identifier subjects for which at least one non-whitespace character survives in any protected mention. Covering only part of a name does not protect it. Over-redaction measures the fraction of gold non-PII, non-whitespace characters removed. Both should be low. Neither measures an attacker’s actual re-identification success, and character retention does not establish downstream task usefulness. These results use exact character coverage from the recorded operators. Metric contract

We compare two separate baselines: a Presidio-based deployment baseline, with its pinned recognizers, threshold and native replacement policy, and stock Presidio, using the benchmark’s English engine and exact local deletion operator. The deployment baseline has English and Dutch pipelines and falls back to Dutch for the other six languages. Stock Presidio uses its English engine throughout. These are specific frozen pipelines; the comparison does not represent every possible Presidio configuration or certify parity with a live deployment. Deployment-baseline specification, gateway operators

Residual leakage on every language

All cells below are percentages [95% Wilson CI], with the same 4,506 identifier subjects in 1,000 conversations per cell. The gateway-trained mDeBERTa-280m has three independent training seeds (0, 1, 2); the gateway-trained MoE has one scored seed (0) in the merged evidence used here. Each baseline has one inference run per configuration. Seeds and languages are never pooled into a larger sample. These intervals describe each run, not uncertainty across training seeds.

The 280m passes the disjoint-Wilson and paired privacy checks in all 24 seed-language comparisons against each baseline. Its paired over-redaction deltas also pass the earlier, stricter no-increase rule in all those comparisons. This is numerical evidence on the frozen test split; missing independent review prevents a scoped contract win. English is its worst language for leakage in every seed, as the table shows. Three-seed encoder scorecard

The MoE’s worst leakage is Romanian, 14.96% [13.95, 16.03], and its lowest is Polish, 9.76% [8.93, 10.67]. Those are descriptive slices of the full matrix, not substitutes for the other languages or a replicated architecture comparison. Its single seed passes the numerical privacy checks against both Presidio baselines, but cannot establish three-seed robustness. MoE scorecard, paired comparisons

How much harmless text was removed

All cells are percentages [95% conversation-bootstrap CI], resampling the same 1,000 conversations per language, including negative examples. The non-PII character denominators are RO 550,519; EN 576,082; DE 591,633; FR 558,749; ES 567,808; IT 572,586; NL 558,344; PL 548,750. These are character counts, not independent statistical samples.

The tables use the merged encoder artifact for its three seeds and baseline intervals, and the merged MoE artifact for seed 0. All bootstrap intervals use 2,000 whole-conversation resamples: RNG seed 125 for the encoder report and 100 for the MoE report. Baseline point estimates agree with the original merged gateway evaluation; bootstrap endpoints can differ between reports. Comparisons use each report’s own paired draws. Wilson intervals describe subject counts; the paired cluster intervals account for subjects sharing a conversation. Encoder estimates and deltas, MoE estimates and deltas, original gateway baselines

The encoder and MoE use their recorded full-conversation windowing and local masking operators. Comparator receipts retain their original runtimes and operators; comparisons in the wider MoE report also retain different window sizes. This post does not claim a controlled head-to-head architecture experiment. Full-split gateway entity F1 is unavailable because some gold spans collide on the whitespace-token grid. All conversations remain in the residual-leak and over-redaction scores above. Encoder protocol and limits, MoE scoring protocol

The losses belong beside the improvements

The MoE over-redacts Polish and German. On Polish gateway conversations it removes 11.58% [10.38, 12.82] of harmless characters; on German, 5.13% [4.55, 5.74]. Both are below the two Presidio baselines in the table, so the contract loss must be attributed precisely: other registered comparators do better on utility. Against the earlier fine-tuned MoE, seed 0’s paired utility delta is +6.65 percentage points [5.83, 7.50] on Polish and +2.40 [1.98, 2.81] on German (1,000 conversation clusters per language, one checkpoint on each side). Both intervals lie wholly above the v3 allowance. The merged report records loss under both the earlier policy and v3. Paired MoE comparisons

Strict-utility calibration failed. A separate, earlier MoE experiment selected confidence thresholds on development data, then froze them before test scoring. Requiring no worse point utility against every comparator during selection reduced recall. On the Romanian gateway test, residual leakage was 56.28% [54.83, 57.72], 65.40% [64.00, 66.78] and 46.78% [45.33, 48.24] for seeds 0, 1 and 2 respectively (4,506 subjects, 1,000 conversations per seed; 95% Wilson CIs). It also broke the legacy Romanian and Polish national-ID zero-leak anchors. This was a recorded loss, not a successful calibration of the gateway-trained candidate. Frozen calibration protocol, three-seed calibration results

The 280m regresses outside gateway traffic. On TAB’s English legal documents, its all-identifier residual leak is 66.91% [65.43, 68.36], 83.18% [81.98, 84.31] and 85.88% [84.76, 86.93] for seeds 0, 1 and 2, versus stock Presidio’s 30.92% [29.50, 32.38] (127 documents, 3,965 identifier subjects per model; 95% Wilson CIs). This is the all-identifier endpoint, distinct from historical TAB DIRECT-only scores. The scorecard also records seed-1 privacy regressions against the earlier encoder on the six general-text configurations, a Polish all-PII regression for seed 2, and an English utility regression against stock Presidio for seed 2. General-text contamination and the capped inference used on these non-gateway sets limit interpretation, but the regressions remain in the report. Encoder results and limitations

Why contract v3 changed prospectively

The earlier utility gate required the upper bound of the paired 95% CI for candidate-minus-comparator over-redaction to be at most zero. That can favor a detector that removes almost nothing, including too little private information. The failed development-only calibration made that tradeoff concrete.

Contract v3 allows at most +1.0 percentage point of additional harmless-character removal at the upper bound of that paired interval, separately for every configuration, comparator and candidate seed. This is an absolute margin of 0.01, not a relative 1% increase. It is a declared engineering tolerance, not evidence that users’ tasks tolerate that loss. Disjoint privacy CIs, negative paired leakage deltas, the Romanian/Polish zero-leak anchors, at least three training seeds, reviewed and disjoint gold, and load requirements remain unchanged. Prospective registration and rationale

The registration records the decision on October 11 before the gateway-trained MoE and encoder evaluations. It applies only to new candidate evaluations after the introducing commit; historical predictions and verdicts cannot be retrospectively promoted. The MoE scoring protocol pins that registration. The merged encoder scorecard still cites the older contract and passes its stricter zero-margin numerical utility check; we retain that provenance rather than relabelling its run as v3. The earlier calibration remains a loss, and the gateway-trained MoE still fails v3. Contract registration, MoE protocol, encoder scorecard

Independent gold review, semantic disjointness, downstream task utility, deployed operator equivalence and end-to-end CPU latency/load acceptance remain unverified. The next work is to resolve those gaps, reduce the MoE’s over-redaction, and preserve the encoder’s quality outside gateway traffic. The roadmap tracks that work. The models remain private while these results and their limits are available for scrutiny.

The protector test, on real legal gold

TAB marks every identifier as DIRECT (directly re-identifying — a name, a case number) or QUASI (needs combination). We score what a de-identifier is for: the DIRECT-identifier leak rate — the fraction of distinct direct identifiers left un-redacted in the residual. Lower is better. Here are the six models on our board, ranked by that leak rate, with their detection-F1 beside it:

Read the F1 column against the leak column. Presidio tops detection-F1 (0.589) and still leaks half of all direct identifiers. GLiNER2 is second on F1 (0.545) and second-worst on leakage (65.1%). The ranking you’d quote from a model card has almost nothing to do with who protects the person. And note our own shipped 280M is last here: it was trained to nail structured national-IDs on synthetic text — a different job than catching names and case numbers in real legal prose. On real judgments, every one of these models leaves roughly half to two-thirds of the direct identifiers in place.

Except one

Train a model on real legal document structure instead — real CJEU judgment layouts (a different court from ECHR, so this is genuine zero-shot, with zero document overlap) with synthetic, checksum-valid identifiers spliced in — and the picture inverts:

9.5% versus the field’s best of 49.6% — it leaves ~5× fewer direct identifiers behind. And it does this while sitting mid-table on detection-F1 (0.340). The mechanism is the whole point: it reliably touches the identifier tokens — redacting them — even when it disagrees with the gold on the exact span boundary or the entity type. Strict detection-F1 punishes that disagreement; a person’s privacy does not care about it.

We checked the obvious cheat — is it just redacting everything? No: it marks 9.1% of tokens as PII while the gold marks 11.3% (it under-redacts overall), with only 4.3% false-positive redaction. It is not blunt; it is aimed.

What is, and isn’t, a claim here

This one we’re willing to lean on, within its scope:

• It’s robust. Three independent training seeds give DIRECT-leak {9.5%, 9.9%, 9.9%} — a 0.4-point spread, nowhere near the gap to the field.
• It’s significant. A paired bootstrap over the 264 shared direct-identifier subjects puts the gap to the runner-up at Δ = −40 points, 95% CI [−47.7, −32.2] — comfortably below zero.
• It’s on real, external, peer-reviewed gold (TAB ECHR), contamination-free.

And the honest boundaries:

• We do not top detection-F1. On TAB, synthetic training caps our detection-F1 around 0.34, well short of Presidio’s 0.589. We’re not claiming the detection crown; we’re claiming the protection one.
• One board, one domain, one language (English legal) so far. Extending the re-id win across languages and domains is the next step, not a done deal.
• Re-identification leak-rate is the metric we lead with — competitors report detection-F1. But it isn’t a metric we invented to flatter ourselves: it’s computed directly from TAB’s own externally-annotated DIRECT/QUASI labels.
• The winning checkpoint isn’t published yet; the numbers above are reproducible from the benchmark and our scorecard, and the model will follow.

The takeaway is the same as last time, now on real court text and with the protector identified: if you choose a de-identification model by its detection-F1, you can choose one that leaks half the identifiers you were trying to protect. Train for structure, score for leakage, and lead with the number that measures the harm.

→ See the figures and provenance on the leaderboard · read the EuroPriv-Bench paper.

This week we added a second, independent diagnostic that looks at a different kind of residue: the quasi-identifiers left behind after redaction — combinations of attributes like a date, a place, and a role that, taken together, can make a record stand out even when every named entity is gone.

A k-anonymity-violation diagnostic, within the corpus

The new diagnostic is a within-corpus k-anonymity-violation measure over the residual quasi-identifiers in redacted text. It flags records whose surviving attribute combination is rare enough that the record is distinctive within the sample — a signal that redaction removed the obvious identifiers but not the quieter, combinable ones.

It’s worth being precise about what this is, because the words matter here:

• It measures sample distinctiveness — how unusual a record is within this corpus.
• It is not population re-identification, and we don’t report it as such. Saying a record is rare in a dataset of a few hundred documents says nothing, on its own, about whether a real person could be picked out of a national population.
• We reserve the term re-identification for the deterministic national-ID channel. For this diagnostic, the honest word is residual distinctiveness.

Why add a second channel at all

The national-ID metric catches the sharpest case: a structured string that, alone, discloses who someone is. But good redaction can clear every national ID and still leave a record that’s distinctive on its quasi-identifiers. A second, independent channel lets us see that residue instead of assuming the structured-ID metric covers it. Two channels that measure different failure modes are harder to fool than one.

This is groundwork. The diagnostic is dev, intended as an internal sensitivity signal — not a validated, citable, or population-level re-identification claim. A separate strand of work (a population-uniqueness estimator with a reference-population module) is what would eventually let us reason about real-world uniqueness, and that is pending a census-calibrated generator — explicitly not something we’re claiming today.

→ See where this fits on the roadmap.

A synthetic eval that everyone aces isn’t an eval

Our own synthetic detection sets were doing their job for span discipline — checksum-valid IDs, offset-correct gold — but they had a quieter problem: on the cleanest configs, the detection scores had saturated. A control run sat at entity-F1 1.000. When the strongest baselines all crowd the ceiling, the metric has stopped discriminating between them, and any ranking you read off it is noise.

So we adopted the Ai4Privacy open-core (openpii-1m, CC-BY-4.0) as an external, independent detection eval — text we didn’t author, scored through the same harness and the same GDPR-aligned taxonomy crosswalk. It de-saturates immediately: across the same models, entity-F1 now spreads over a 0.41–0.67 band instead of pinning at 1.000. That spread is the point — the eval now separates models that the synthetic control could not.

One licensing note, because it’s the kind of thing that’s easy to get wrong: Ai4Privacy ships a larger 500k tier that is Llama-licensed, which fails our cleanly-licensed-only gate. We caught it and excluded that tier; only the CC-BY-4.0 open-core feeds the eval. A benchmark that quietly ingests non-redistributable data can’t claim to be open.

Real legal data is harder than our synthetic — and we’d rather show it

The bigger step: we integrated TAB (the Text Anonymization Benchmark, English legal text derived from European Court of Human Rights judgments, MIT-licensed) as the first real-data gold config in the suite. Everything in EuroPriv-Bench until now was synthetic — clean by construction. TAB is human-written, human-annotated court text: 127 documents, projected into our taxonomy with zero misaligned spans after alignment. It carries a new status, config_status = real-external-gold, to mark it as exactly that: real, externally-sourced, not one of our own synthetic skeletons.

The honest finding is that real legal English is hard, and it humbles our own model:

Read that the right way. The strongest system on this real-legal config is Presidio, at 0.589 — far below the near-ceiling numbers everyone posts on synthetic data. And our own kp-deid sits at 0.199: it was trained on Romanian, and on real English legal text it is out of distribution and struggles. We are not the leader here, and the table says so. That’s the value of a real-data config — it shows where the synthetic-trained model doesn’t transfer, instead of flattering it.

Status, said plainly

Both additions are config_status = dev for the Ai4Privacy detection eval and real-external-gold for TAB — measured, contamination-controlled signals, not validated, citable, or SOTA claims. The TAB-derived gold config is private for now (licensing/redistribution review pending), so there’s no public config to load yet — the numbers above are reproducible internally against the committed harness, and we’ll say so on every row.

None of this changes the metric we lead with. Detection-F1 is a detection number; the re-identification-risk metric still leads on the national-ID tracks, where a missed ID decodes to a birthday, a sex, and a county. What this week did was make the detection side honest: an external eval that separates models, and a real-data config that refuses to let our own model off easy.

→ See the public configs and their provenance on the leaderboard, or read the roadmap for where the real-data and external-eval work sits.

We lead with re-identification risk, not detection-F1. A missed national ID isn’t one missed token: a Romanian CNP decodes to a birthday, a sex, and a county at once. So the question the board asks first is not how many entities did you label correctly — it’s what fraction of subjects had their national ID slip through.

What the three external systems are — and aren’t

None of these tools was built to model national-ID structure. They do ordinary named-entity recognition, and they do it as designed; they were never tuned for the re-identification metric the board leads with. That’s exactly why they’re a useful, honest baseline.

• spaCy en_core_web_lg (Explosion) — a general-purpose English NER pipeline.
• GLiNER2 gliner2-base-v1 (Fastino) — a zero-shot span tagger you point at PII labels.
• Presidio (presidio-analyzer + en_core_web_lg, Microsoft) — a PII framework that pairs NER with deterministic recognizers (regexes, checksums) for structured IDs.

The numbers, on contamination-free Romanian

The ro-realskeleton-v1 track scores each model on 1,123 distinct CNP subjects in held-out, source-separated legal text. We report CNP leak-rate: the fraction of subjects whose national ID slips through.

Read the spread. spaCy leaks roughly 89% of CNPs — its general NER never modelled the 13-digit structure, so the IDs walk straight through (3,000 quasi-identifiers exposed). GLiNER2 leaks ~29%, better but still roughly one CNP in three. Presidio leaks 0% — not because it out-detects the others on F1 (it sits mid-table at 0.472), but because its deterministic recognizers catch the national-ID string whatever the surrounding NER does. Structure-aware coverage, not detection-F1, is what protects the person — and that’s the case the board is built to make.

For contrast, our own kp-deid-mdeberta-280m also leaks 0%, at a higher F1 (0.741) — but the point of this post is the three external rows, not the in-house one.

An honest caveat about status

These real-skeleton tracks are in development (config_status = dev): built from authored template families and not yet native-speaker- or inter-annotator-agreement-validated. Treat the figures above as measured, contamination-controlled signals — not validated, citable, or SOTA claims. The leaderboard marks every such row dev for exactly this reason, and validation comes before we lean on any of it.

We’re also not the only people pointing at re-identification risk: concurrent, independent work such as RAT-Bench builds a re-identification-risk leaderboard over synthetic U.S. demographics. EuroPriv-Bench’s angle is the underserved part — European legal text, a GDPR-aligned taxonomy, and national-ID structure decoded directly. We think it’s the first unified pan-European benchmark to put detection and re-identification risk on the same gold; we don’t claim to be first to care.

Bring your model

The submission CI is open and runs without secrets — same harness, same provenance, same dev/ citable discipline for everyone. If you maintain a de-identification model, add your row.

→ See every figure with its provenance on the leaderboard · open a submission via How to submit · read the EuroPriv-Bench paper.

Detection-F1 and re-identification protection come apart. A model can score high on entity-F1 and still leak the identifiers that re-identify a person; a model trained for protection can sit mid-table on F1 and leak essentially nothing. The two numbers measure different things, and on national IDs they disagree hard.

The dissociation, on contamination-free Romanian

The ro-realskeleton-v1 track scores each model on 1,123 distinct CNP subjects — Romanian national IDs in held-out, source-separated legal text that no baseline has trained on. We report CNP leak-rate: the fraction of subjects whose national ID slips through. A single missed CNP is not one missed token — it decodes to a birthday, a sex, and a county at once, so we count what a miss exposes (the “quasi-identifiers leaked” column below).

Read the top two rows together. GLiNER has the highest detection-F1 on the board (0.853) and the second-worst leak-rate (30.2%) — it is type-accurate on the entities it catches, yet roughly one CNP in three walks straight through. kp-deid scores lower on F1 (0.741) and leaks 0% of 1,123 CNPs. tabularisai tells the same story from the other direction: a strong-looking 0.747 F1, the worst leak-rate on the board (35.4%). F1 simply does not predict who protects the person.

Why the gap? F1 rewards getting the label right; protection only needs coverage — redact the 13-digit string and no harm is done, whatever you call it. A detector tuned to label precisely can still drop the spans that matter most, and detection-F1 will not flag it. (openai/privacy-filter is the instructive corner case: weak F1, but it over-redacts, so its leak-rate is low too.)

A preliminary cross-language signal: Polish PESEL, zero-shot

The same dissociation shows up in a second language and a second identifier — zero-shot. kp-deid was trained on Romanian and never saw Polish, yet on the pl-realskeleton-v1 track (1,096 distinct PESEL subjects, the Polish national ID, which carries its own structure and check digit) it still leaks nothing:

GLiNER again leads on the structural F1 you’d quote in a model card (0.825) and leaks more than half of all PESELs (57.8%). A model that was never shown Polish protects the Polish ID; a higher-F1 multilingual detector leaks the majority of them.

An honest caveat, because it matters. This Polish track is early. It is still in development (dev), built from a single authored template family, and not yet native-speaker- or IAA-validated. Treat it as a strong early signal that the dissociation generalizes across languages and identifier schemes — not as a validated, citable headline. Full validation comes before we lean on it.

What is and isn’t a claim here

The Romanian real-skeleton track is itself still in development (dev), pending native-speaker and inter-annotator-agreement validation. So the result above is a measured, contamination-controlled delta on held-out data — not yet a peer-reviewed, citable claim. We’d rather say that plainly than round it up.

We’re also not the only people pointing at re-identification risk: concurrent, independent work like RAT-Bench builds a re-identification-risk leaderboard over synthetic text on U.S. demographics. EuroPriv-Bench’s angle is the part that’s underserved — European legal text, a GDPR-aligned taxonomy, and national-ID structure decoded directly. We think it’s the first unified pan-European benchmark to put detection and re-identification risk side by side on the same gold; we don’t think it’s the first to care about the problem.

The takeaway is narrow and, we think, durable: if you pick a de-identification model on detection-F1, you can pick one that leaks the identifiers you were trying to protect. Lead with the leak-rate.

→ See every figure above, with its provenance, on the leaderboard · read the EuroPriv-Bench paper · get the model: klusai/kp-deid-mdeberta-280m.

• klusai/ds-kp-general-ro-50k — Romanian
• klusai/ds-kp-general-en-50k — English
• klusai/ds-kp-general-pl-50k — Polish

These aren’t the first synthetic-PII datasets — the field has plenty. They’re the first datasets in the EuroPriv-Bench suite, and they’re built to a quality bar most synthetic-PII sets miss.

The bar: valid IDs, correct spans, honest provenance

Synthetic PII is easy to generate and easy to get subtly wrong. We held these sets to four properties, each one checkable:

• Checksum-valid locale identifiers. Where an identifier carries a check digit, the emitted value passes it: Romanian CNP, Polish PESEL and NIP, IBAN (mod-97), payment cards (Luhn). Invalid or fake-checksum IDs are not emitted. A benchmark that quietly ships malformed national IDs can’t tell you whether a model handles real ones.
• Offset-correct gold spans, by construction. Because the PII is injected into the text, we know exactly where every entity sits — no post-hoc span alignment to drift. We verify it anyway: 100% byte-equality between each gold span and the substring it points at, and 100% BIOES validity on the tag sequence.
• Zero train/gold overlap. Training material and gold are kept strictly apart, so a model can’t score on rows it has effectively seen.
• Uniform provenance, per dataset. Each card records source and license, the generator seed, and the taxonomy version — the same fields, the same shape, every time.

Cleanly-licensed, openly redistributable

The whole set is CC-BY. We sourced cleanly-licensed material only, so the datasets are openly redistributable end to end — no “results you can see but data you can’t.” That’s a precondition for the kind of reproducibility EuroPriv-Bench is built around: a number you can’t trace to data you can load isn’t one you can check.

How it fits the program

These sets feed the EuroPriv-Bench models and benchmark — the general-domain foundation the detection and re-identification-risk evaluations run on. Romanian matters here in particular: it isn’t in the common training corpora, so a Romanian gold set tests models on genuinely unseen ground.

Re-identification risk, not detection-F1, is the metric we lead with — a missed national ID isn’t one missed token, it’s a leaked birthday, sex, and county at once. Clean, checksum-valid IDs are what make that measurement honest: you can only count what a miss exposes if the ID you’re decoding is structurally real.

This is the general-domain bring-up. Legal and clinical synthesis come next.

→ See where these feed in on the leaderboard, or read the roadmap.

The gap it exposes

Models advertising 96–97% F1 on English PII benchmarks tell you little about how they behave on, say, Dutch clinical notes under a European privacy taxonomy. When we hold current baselines to that bar, F1 drops to 0.44–0.61 across en/fr/es/de/it/nl — with recall (the privacy-critical failure mode) the weakest link.

→ See the live leaderboard

What’s in v0

• 6 European language configs from cleanly-licensed open sources, remapped to the unified KlusAI privacy taxonomy and fully attributed.
• A reproducible harness with provenance baked into every result (model id, dataset config/split, harness & taxonomy version, timestamp).
• Baselines for the current public de-identification models.

What’s next

More baselines (multilingual de-id models, GLiNER, Presidio), larger sample sizes, and the legal/clinical splits — plus under-served languages including Romanian. Follow along here or on GitHub.

EuroPriv-Bench is open — see How to submit.

The model under test is openai/privacy-filter: a sparse Mixture-of-Experts with only 50M active parameters, doing short-sequence token classification. We measured inference per example, several ways:

Two surprises:

1. The GPU lost. Both the PyTorch-MPS and the MLX paths were slower than CPU. With so few active parameters and short sequences, there isn’t enough work per item to amortize the overhead of getting data onto the GPU — and the MoE’s routing ops fall off the Metal fast path and bounce back to the CPU anyway. GPUs win on big compute (large models, long sequences, training); they don’t on a tiny MoE doing short spans.

2. More threads were slower. PyTorch’s default (28 BLAS threads here) ran at half the speed of 4 threads — the ops are small enough that thread-coordination overhead dominates.

What actually used the machine

The real lever was parallelism at the job level: run many small, thread-capped jobs at once. Seven worker processes × 4 threads saturates the 28 cores, each job at its own fastest point. A full sweep (multiple models × multiple language configs) that crawled before now finishes in a couple of minutes — a ~7× wall-clock win, and identical (deterministic) numbers.

The lesson

Profile before you reach for the accelerator. “Use the GPU” and “use all the cores” are heuristics, not laws — and for small models on short inputs, both can cost you. The GPU still earns its keep here: just for the other job, training our own models, where the compute is actually large.