Flagship benchmark

EuroPriv-Bench Leaderboard

Detection ≠ protection: on realistic Romanian documents the highest-F1 detector still leaks ~30% of national IDs (CNPs), while the model that protects best leaks 0%.

For privacy, the number that matters is re-identification leakage — how many decode-bearing national IDs a model leaves un-redacted — not detection F1. A leaked ID silently discloses identifying attributes (a Romanian CNP encodes date of birth, sex and county), so the best detector is not necessarily the best protector. Everything below is dev (pending native-speaker / inter-annotator-agreement sign-off) — read it as a strong early signal, not a validated, citable result.

8 models 3 decode-bearing national IDs (CNP · PESEL · Codice Fiscale) 3 leakage tracks: RO · PL · IT detection across 8 languages re-id leak rate 0%–96%

Scatter plot of detection F1 (x-axis) against CNP re-identification leak rate (y-axis) for eight models on ro-realskeleton-v1; higher F1 does not imply lower leakage. — Detection–protection dissociation: detection F1 (x) vs CNP re-identification leak rate (y) on `ro-realskeleton-v1`, `dev` split, n=1123 CNPs. Higher F1 does not imply lower leakage.

Re-identification leakage — the metric that matters

Detection F1 is not privacy. EuroPriv-Bench measures re-identification leakage: a missed (un-redacted) national ID deterministically discloses identifying attributes — on the Romanian configs a leaked CNP discloses date of birth + sex + county, on the Polish track a leaked PESEL discloses date of birth + sex, and on the Italian track a leaked Codice Fiscale discloses date of birth + sex + place of birth. The bar shows the leak rate (long/red = leaks, tiny/green ≈ protects); the table also counts national IDs left un-redacted and the quasi-identifiers thereby leaked (lower is better).

Model	Track	Contamination	Validation	Leak rate	95% CI	IDs missed	Quasi-identifiers
`fastino/gliner2-base-v1`	`legal-realskeleton-v1`	clean held-out	dev	65.4%	63.0–67.8	981	2943
`fastino/gliner2-base-v1`	`ro-realskeleton-v1`	clean held-out	dev	28.6%	26.0–31.3	321	963
`fastino/gliner2-base-v1`	`ro-synthetic-v1`	clean held-out	dev	8.1%	6.5–9.9	82	246
`fastino/gliner2-base-v1`	`it-realskeleton-v1`	clean held-out	dev	39.2%	36.3–42.1	435	1305
`fastino/gliner2-base-v1`	`pl-realskeleton-v1`	clean held-out	dev	33.8%	31.0–36.6	370	740
`urchade/gliner_multi_pii-v1`	`legal-realskeleton-v1`	clean held-out	dev	71.7%	69.4–74.0	1076	3228
`urchade/gliner_multi_pii-v1`	`ro-realskeleton-v1`	clean held-out	dev	30.2%	27.6–32.9	339	1017
`urchade/gliner_multi_pii-v1`	`ro-synthetic-v1`	unknown	dev	0.0%	0.0–0.4	0	0
`urchade/gliner_multi_pii-v1`	`it-realskeleton-v1`	clean held-out	dev	36.2%	33.4–39.1	402	1206
`urchade/gliner_multi_pii-v1`	`pl-realskeleton-v1`	clean held-out	dev	57.8%	54.9–60.7	634	1268
`klusai/kp-deid-mdeberta-280m`	`legal-realskeleton-v1`	clean held-out	dev	4.1%	3.2–5.2	61	183
`klusai/kp-deid-mdeberta-280m`	`ro-realskeleton-v1`	clean held-out	dev	0.0%	0.0–0.3	0	0
`klusai/kp-deid-mdeberta-280m`	`ro-synthetic-v1`	in-distribution	dev	0.0%	0.0–0.4	0	0
`klusai/kp-deid-mdeberta-280m`	`it-realskeleton-v1`	clean held-out	dev	0.0%	0.0–0.3	0	0
`klusai/kp-deid-mdeberta-280m`	`pl-realskeleton-v1`	clean held-out	dev	0.0%	0.0–0.3	0	0
`klusai/kp-deid-xlmr-560m`	`legal-realskeleton-v1`	clean held-out	dev	0.0%	0.0–0.3	0	0
`klusai/kp-deid-xlmr-560m`	`ro-realskeleton-v1`	clean held-out	dev	0.0%	0.0–0.3	0	0
`klusai/kp-deid-xlmr-560m`	`ro-synthetic-v1`	in-distribution	dev	0.0%	0.0–0.4	0	0
`klusai/kp-deid-xlmr-560m`	`pl-realskeleton-v1`	clean held-out	dev	0.0%	0.0–0.3	0	0
`OpenMed/privacy-filter-multilingual`	`legal-realskeleton-v1`	clean held-out	dev	83.9%	81.9–85.6	1258	3774
`OpenMed/privacy-filter-multilingual`	`ro-realskeleton-v1`	clean held-out	dev	26.4%	23.9–29.0	296	888
`OpenMed/privacy-filter-multilingual`	`ro-synthetic-v1`	unknown	dev	1.9%	1.2–2.9	19	57
`OpenMed/privacy-filter-multilingual`	`it-realskeleton-v1`	clean held-out	dev	35.4%	32.6–38.2	393	1179
`OpenMed/privacy-filter-multilingual`	`pl-realskeleton-v1`	clean held-out	dev	3.7%	2.8–5.0	41	82
`presidio-analyzer+en_core_web_lg`	`legal-realskeleton-v1`	clean held-out	dev	66.5%	64.0–68.8	997	2991
`presidio-analyzer+en_core_web_lg`	`ro-realskeleton-v1`	clean held-out	dev	0.0%	0.0–0.3	0	0
`presidio-analyzer+en_core_web_lg`	`ro-synthetic-v1`	clean held-out	dev	0.0%	0.0–0.4	0	0
`presidio-analyzer+en_core_web_lg`	`it-realskeleton-v1`	clean held-out	dev	96.5%	95.2–97.4	1072	3216
`presidio-analyzer+en_core_web_lg`	`pl-realskeleton-v1`	clean held-out	dev	0.0%	0.0–0.3	0	0
`openai/privacy-filter`	`legal-realskeleton-v1`	clean held-out	dev	96.7%	95.6–97.5	1450	4350
`openai/privacy-filter`	`ro-realskeleton-v1`	clean held-out	dev	1.4%	0.9–2.3	16	48
`openai/privacy-filter`	`ro-synthetic-v1`	unknown	dev	0.1%	0.0–0.6	1	3
`openai/privacy-filter`	`it-realskeleton-v1`	clean held-out	dev	0.4%	0.1–0.9	4	12
`openai/privacy-filter`	`pl-realskeleton-v1`	clean held-out	dev	0.1%	0.0–0.5	1	2
`spacy/en_core_web_lg@3.8.0`	`legal-realskeleton-v1`	clean held-out	dev	66.0%	63.6–68.4	990	2970
`spacy/en_core_web_lg@3.8.0`	`ro-realskeleton-v1`	clean held-out	dev	89.0%	87.1–90.7	1000	3000
`spacy/en_core_web_lg@3.8.0`	`ro-synthetic-v1`	clean held-out	dev	91.0%	89.0–92.6	925	2775
`spacy/en_core_web_lg@3.8.0`	`it-realskeleton-v1`	clean held-out	dev	39.2%	36.3–42.1	435	1305
`spacy/en_core_web_lg@3.8.0`	`pl-realskeleton-v1`	clean held-out	dev	44.7%	41.8–47.7	490	980
`tabularisai/eu-pii-safeguard`	`legal-realskeleton-v1`	clean held-out	dev	27.3%	25.1–29.6	409	1227
`tabularisai/eu-pii-safeguard`	`ro-realskeleton-v1`	clean held-out	dev	35.4%	32.6–38.2	397	1191
`tabularisai/eu-pii-safeguard`	`ro-synthetic-v1`	unknown	dev	0.0%	0.0–0.4	0	0
`tabularisai/eu-pii-safeguard`	`it-realskeleton-v1`	clean held-out	dev	34.4%	31.6–37.2	382	1146
`tabularisai/eu-pii-safeguard`	`pl-realskeleton-v1`	clean held-out	dev	30.7%	28.0–33.5	336	672

The dissociation is the point: on realistic-structure Romanian documents (ro-realskeleton-v1) the model with the best detection F1 leaks ~30% of CNPs, while a purpose-built protector redacts every one. The same pattern repeats zero-shot on the Polish PESEL and Italian Codice Fiscale tracks. The mechanism is general — aggregate detection F1 can stay high while a model misses the rare, high-stakes tokens that carry the re-identification — and decode-bearing national identifiers (RO CNP, PL PESEL, IT codice fiscale) are the clearest, provable case of it, which is why this benchmark leads with leakage. Extending the measure to quasi-identifier-combination re-identification is in progress, so the broad reading is a hypothesis under test rather than a settled law. All tracks are still dev (pending native-speaker / inter-annotator-agreement validation) — read their leak rates as strong early signals, not yet validated headline results.

Detection scores — by model and language

Entity-level scores on the klusai/europriv-bench test split, by model and language. Higher F1 is better; the table defaults to best-first. Click a column header to re-sort. Rows where the model was trained on the config’s own source data are greyed (in-distribution) — their scores are inflated by train/eval overlap and are not a fair test.

How to read this — contamination & validation

Each row carries two governance markers. Contamination flags whether the model was trained on that config's source data — an in-distribution score is inflated by train/eval overlap (e.g. a perfect 100/100/100 is a memorisation artefact, not a win), while a clean held-out score is a fair test. Validation shows whether a config has passed native-speaker / inter-annotator-agreement (IAA) sign-off: only a citable row may be cited as a validated result. Everything is currently dev — not yet citable.

Each row reports entity-level precision / recall / F1 (×100) under the unified KlusAI privacy taxonomy. Results carry full provenance (model id, dataset config/split, harness & taxonomy version, timestamp) in the source repository.

Schema v3 · Benchmark v0.2.0 · Taxonomy v0.2.0

Model	Adapter	Lang	Domain	Precision	Recall	F1	n	Contamination	Validation
`fastino/gliner2-base-v1`	gliner2	ro	legal	82.5	55.9	66.7	1500	clean held-out	dev
`fastino/gliner2-base-v1`	gliner2	ro	legal	74.6	56.4	64.2	1500	clean held-out	dev
`fastino/gliner2-base-v1`	gliner2	ro	general	80.2	77.3	78.7	1500	clean held-out	dev
`fastino/gliner2-base-v1`	gliner2	it	legal	87.3	69.2	77.2	1500	clean held-out	dev
`fastino/gliner2-base-v1`	gliner2	pl	legal	57.4	47.4	51.9	1500	clean held-out	dev
`fastino/gliner2-base-v1`	gliner2	nl	general	50.7	42.8	46.4	1500	clean held-out	dev
`fastino/gliner2-base-v1`	gliner2	en	general	58.4	45.3	51.0	1500	clean held-out	dev
`fastino/gliner2-base-v1`	gliner2	fr	general	56.6	47.6	51.7	1500	clean held-out	dev
`fastino/gliner2-base-v1`	gliner2	de	general	56.1	47.1	51.2	1500	clean held-out	dev
`fastino/gliner2-base-v1`	gliner2	it	general	54.6	42.8	48.0	1500	clean held-out	dev
`fastino/gliner2-base-v1`	gliner2	es	general	56.8	47.6	51.8	1500	clean held-out	dev
`urchade/gliner_multi_pii-v1`	gliner	ro	legal	89.3	72.9	80.3	1500	clean held-out	dev
`urchade/gliner_multi_pii-v1`	gliner	ro	legal	93.9	78.2	85.3	1500	clean held-out	dev
`urchade/gliner_multi_pii-v1`	gliner	ro	general	82.2	80.4	81.3	1500	unknown	dev
`urchade/gliner_multi_pii-v1`	gliner	it	legal	92.1	80.0	85.6	1500	clean held-out	dev
`urchade/gliner_multi_pii-v1`	gliner	pl	legal	93.6	73.7	82.5	1500	clean held-out	dev
`urchade/gliner_multi_pii-v1`	gliner	nl	general	63.9	50.9	56.7	1500	unknown	dev
`urchade/gliner_multi_pii-v1`	gliner	en	general	61.9	42.0	50.0	1500	unknown	dev
`urchade/gliner_multi_pii-v1`	gliner	fr	general	65.6	48.7	55.9	1500	unknown	dev
`urchade/gliner_multi_pii-v1`	gliner	de	general	64.1	51.7	57.2	1500	unknown	dev
`urchade/gliner_multi_pii-v1`	gliner	it	general	64.5	46.6	54.1	1500	unknown	dev
`urchade/gliner_multi_pii-v1`	gliner	es	general	66.2	47.4	55.2	1500	unknown	dev
`klusai/kp-deid-mdeberta-280m`	kp-model	ro	legal	51.5	71.8	60.0	1500	clean held-out	dev
`klusai/kp-deid-mdeberta-280m`	kp-model	ro	legal	68.6	80.5	74.1	1500	clean held-out	dev
`klusai/kp-deid-mdeberta-280m`	kp-model	ro	general	100.0	100.0	100.0memorised	1500	in-distribution	dev
`klusai/kp-deid-mdeberta-280m`	kp-model	it	legal	62.1	79.7	69.8	1500	clean held-out	dev
`klusai/kp-deid-mdeberta-280m`	kp-model	pl	legal	71.0	82.3	76.3	1500	clean held-out	dev
`klusai/kp-deid-mdeberta-280m`	kp-model	nl	general	57.0	48.4	52.3	1500	clean held-out	dev
`klusai/kp-deid-mdeberta-280m`	kp-model	en	general	51.8	41.9	46.4	1500	unknown	dev
`klusai/kp-deid-mdeberta-280m`	kp-model	fr	general	56.0	46.7	50.9	1500	clean held-out	dev
`klusai/kp-deid-mdeberta-280m`	kp-model	de	general	56.1	46.2	50.7	1500	clean held-out	dev
`klusai/kp-deid-mdeberta-280m`	kp-model	it	general	50.6	41.5	45.6	1500	clean held-out	dev
`klusai/kp-deid-mdeberta-280m`	kp-model	es	general	51.2	42.6	46.5	1500	clean held-out	dev
`klusai/kp-deid-xlmr-560m`	kp-model	ro	legal	50.7	62.4	55.9	1500	clean held-out	dev
`klusai/kp-deid-xlmr-560m`	kp-model	ro	legal	63.1	71.9	67.2	1500	clean held-out	dev
`klusai/kp-deid-xlmr-560m`	kp-model	ro	general	100.0	100.0	100.0memorised	1500	in-distribution	dev
`klusai/kp-deid-xlmr-560m`	kp-model	pl	legal	69.4	78.6	73.7	1500	clean held-out	dev
`klusai/kp-deid-xlmr-560m`	kp-model	nl	general	58.4	53.3	55.7	1500	clean held-out	dev
`klusai/kp-deid-xlmr-560m`	kp-model	en	general	52.7	45.9	49.1	1500	unknown	dev
`klusai/kp-deid-xlmr-560m`	kp-model	fr	general	57.0	51.7	54.2	1500	clean held-out	dev
`klusai/kp-deid-xlmr-560m`	kp-model	de	general	57.6	53.0	55.2	1500	clean held-out	dev
`klusai/kp-deid-xlmr-560m`	kp-model	it	general	54.8	47.6	50.9	1500	clean held-out	dev
`klusai/kp-deid-xlmr-560m`	kp-model	es	general	55.9	50.6	53.1	1500	clean held-out	dev
`OpenMed/privacy-filter-multilingual`	openmed	ro	legal	55.7	44.3	49.3	1500	clean held-out	dev
`OpenMed/privacy-filter-multilingual`	openmed	ro	legal	63.8	52.5	57.6	1500	clean held-out	dev
`OpenMed/privacy-filter-multilingual`	openmed	ro	general	71.8	76.4	74.1	1500	unknown	dev
`OpenMed/privacy-filter-multilingual`	openmed	it	legal	66.5	64.7	65.6	1500	clean held-out	dev
`OpenMed/privacy-filter-multilingual`	openmed	pl	legal	73.1	60.5	66.2	1500	clean held-out	dev
`OpenMed/privacy-filter-multilingual`	openmed	nl	general	69.8	57.6	63.1	1500	in-distribution	dev
`OpenMed/privacy-filter-multilingual`	openmed	en	general	66.3	54.7	59.9	1500	in-distribution	dev
`OpenMed/privacy-filter-multilingual`	openmed	fr	general	66.7	56.3	61.1	1500	in-distribution	dev
`OpenMed/privacy-filter-multilingual`	openmed	de	general	67.0	55.6	60.8	1500	in-distribution	dev
`OpenMed/privacy-filter-multilingual`	openmed	it	general	60.7	50.3	55.0	1500	in-distribution	dev
`OpenMed/privacy-filter-multilingual`	openmed	es	general	66.0	53.5	59.1	1500	in-distribution	dev
`presidio-analyzer+en_core_web_lg`	presidio	ro	legal	42.9	43.9	43.4	1500	clean held-out	dev
`presidio-analyzer+en_core_web_lg`	presidio	ro	legal	47.2	47.3	47.2	1500	clean held-out	dev
`presidio-analyzer+en_core_web_lg`	presidio	ro	general	54.7	55.9	55.3	1500	clean held-out	dev
`presidio-analyzer+en_core_web_lg`	presidio	it	legal	51.5	48.0	49.6	1500	clean held-out	dev
`presidio-analyzer+en_core_web_lg`	presidio	pl	legal	45.5	49.3	47.3	1500	clean held-out	dev
`presidio-analyzer+en_core_web_lg`	presidio	nl	general	22.4	23.1	22.7	1500	clean held-out	dev
`presidio-analyzer+en_core_web_lg`	presidio	en	general	55.8	35.9	43.7	1500	clean held-out	dev
`presidio-analyzer+en_core_web_lg`	presidio	fr	general	35.4	24.2	28.7	1500	clean held-out	dev
`presidio-analyzer+en_core_web_lg`	presidio	de	general	19.8	26.4	22.6	1500	clean held-out	dev
`presidio-analyzer+en_core_web_lg`	presidio	it	general	25.3	22.0	23.6	1500	clean held-out	dev
`presidio-analyzer+en_core_web_lg`	presidio	es	general	28.1	20.6	23.8	1500	clean held-out	dev
`openai/privacy-filter`	privacy-filter	ro	legal	37.5	26.8	31.3	1500	clean held-out	dev
`openai/privacy-filter`	privacy-filter	ro	legal	38.8	34.2	36.3	1500	clean held-out	dev
`openai/privacy-filter`	privacy-filter	ro	general	56.0	59.2	57.6	1500	unknown	dev
`openai/privacy-filter`	privacy-filter	it	legal	71.1	65.4	68.1	1500	clean held-out	dev
`openai/privacy-filter`	privacy-filter	pl	legal	42.6	38.8	40.6	1500	clean held-out	dev
`openai/privacy-filter`	privacy-filter	nl	general	59.3	39.1	47.1	1500	unknown	dev
`openai/privacy-filter`	privacy-filter	en	general	58.0	32.3	41.5	1500	unknown	dev
`openai/privacy-filter`	privacy-filter	fr	general	59.2	38.2	46.4	1500	unknown	dev
`openai/privacy-filter`	privacy-filter	de	general	63.3	41.4	50.0	1500	unknown	dev
`openai/privacy-filter`	privacy-filter	it	general	56.7	37.5	45.1	1500	unknown	dev
`openai/privacy-filter`	privacy-filter	es	general	59.3	38.3	46.5	1500	unknown	dev
`spacy/en_core_web_lg@3.8.0`	spacy	ro	legal	10.9	17.9	13.6	1500	clean held-out	dev
`spacy/en_core_web_lg@3.8.0`	spacy	ro	legal	14.2	14.3	14.3	1500	clean held-out	dev
`spacy/en_core_web_lg@3.8.0`	spacy	ro	general	30.0	13.5	18.6	1500	clean held-out	dev
`spacy/en_core_web_lg@3.8.0`	spacy	it	legal	15.9	14.3	15.1	1500	clean held-out	dev
`spacy/en_core_web_lg@3.8.0`	spacy	pl	legal	16.7	14.7	15.7	1500	clean held-out	dev
`spacy/en_core_web_lg@3.8.0`	spacy	nl	general	11.5	9.9	10.7	1500	clean held-out	dev
`spacy/en_core_web_lg@3.8.0`	spacy	en	general	48.8	23.4	31.7	1500	clean held-out	dev
`spacy/en_core_web_lg@3.8.0`	spacy	fr	general	17.8	8.9	11.9	1500	clean held-out	dev
`spacy/en_core_web_lg@3.8.0`	spacy	de	general	9.4	10.7	10.0	1500	clean held-out	dev
`spacy/en_core_web_lg@3.8.0`	spacy	it	general	11.8	8.4	9.8	1500	clean held-out	dev
`spacy/en_core_web_lg@3.8.0`	spacy	es	general	12.1	6.8	8.7	1500	clean held-out	dev
`tabularisai/eu-pii-safeguard`	tabularisai	ro	legal	77.8	50.3	61.1	1500	clean held-out	dev
`tabularisai/eu-pii-safeguard`	tabularisai	ro	legal	90.1	63.7	74.7	1500	clean held-out	dev
`tabularisai/eu-pii-safeguard`	tabularisai	ro	general	89.3	86.0	87.6	1500	unknown	dev
`tabularisai/eu-pii-safeguard`	tabularisai	it	legal	89.6	61.1	72.7	1500	clean held-out	dev
`tabularisai/eu-pii-safeguard`	tabularisai	pl	legal	86.7	64.3	73.8	1500	clean held-out	dev
`tabularisai/eu-pii-safeguard`	tabularisai	nl	general	71.2	56.4	62.9	1500	in-distribution	dev
`tabularisai/eu-pii-safeguard`	tabularisai	en	general	64.4	42.9	51.5	1500	in-distribution	dev
`tabularisai/eu-pii-safeguard`	tabularisai	fr	general	69.7	51.8	59.4	1500	in-distribution	dev
`tabularisai/eu-pii-safeguard`	tabularisai	de	general	71.7	56.9	63.4	1500	in-distribution	dev
`tabularisai/eu-pii-safeguard`	tabularisai	it	general	68.9	50.0	57.9	1500	in-distribution	dev
`tabularisai/eu-pii-safeguard`	tabularisai	es	general	67.7	51.2	58.3	1500	in-distribution	dev

How to submit

EuroPriv-Bench is open. Run the harness against your model and open a PR adding your entry to baselines/leaderboard.json — see the benchmark repo for the adapter contract and reproduction steps. Entries without reproducible provenance are not listed.