Detection benchmark

Caution

Every number on this page comes from a run we performed on 2026-07-25, on one corpus, with one scorer, against pinned versions of every system. Nothing is estimated, projected, or carried over from an earlier engine. The corpus is synthetic and has known biases — they are listed at the bottom, and they matter. This is a measurement on one corpus, not a general statement about any product.

What was measured

Three PII-detection systems, four configurations, one shared corpus and one shared scorer:

System	Version pin	Configuration
CloakAPI A — engine only	ruleset 1.4.29, WASM sha-256 00a4c7c7ad27fde5…4cfab50, @cloakapi/sdk 1.1.50	The WebAssembly pattern/checksum engine plus its built-in Latin name dictionary. No download, runs immediately.
CloakAPI B — engine + on-device model	same engine, plus enhanced-ner-distilbert-mling-hrl (int8 ONNX, ~135 MB) and the name gazetteer (12 066 given / 13 045 surnames)	The shipped default in the apps. Thresholds exactly as shipped.
Microsoft Presidio	presidio-analyzer 2.2.363 · spaCy 3.8.13 · en_core_web_lg	Out-of-the-box AnalyzerEngine(), nothing disabled, language="en".
OpenAI Privacy Filter	openai/privacy-filter (Apache-2.0), q4f16 ONNX via @huggingface/transformers 4.2.0	Out-of-the-box token-classification pipeline.

Both CloakAPI configurations are published because they are genuinely different systems. Config B is the like-for-like comparison against the two baselines (both of which are model-based). Config A is what a user gets before any model is downloaded.

Corpus

320 texts · 40 languages (17 non-Latin, spanning 12 scripts) · 1340 gold entities. Eight short admin texts per language — clinic intake, bank note, HR record, insurance claim, employer letter, utility account note, referral note, and one running-prose sentence.

class	gold entities
person	320
dob	240
place	160
phone	160
email	146
org	120
card	106
iban	60
govid	28

Everything is synthetic and generated by a seeded, deterministic script: person names are invented; card numbers are Luhn-valid; IBANs are ISO 13616 mod-97-valid; gov-IDs are checksum-valid under 15 national algorithms; dates of birth use five native constructs per language (including Buddhist-era Thai, Shōwa-era Japanese, and Arabic-Indic/Persian/Bengali digits). Place and organisation names are real, public entities whose native-script labels were verified live against Wikidata (organisations additionally checked via their P31 claims so no city is mistaken for a company).

How a hit is counted

• Recall is entity-level and span-overlap based. A gold entity counts as detected-any if any predicted span overlaps its character range, and detected-typed if an overlapping span also carries the same class. Overlap matching means a system is not penalised for chunking spans differently.
• Precision is span-level. precision-any = predicted spans that overlap some gold entity ÷ all predicted spans. precision-typed(T) = spans predicted as class T that overlap a gold T ÷ all spans predicted as T. The corpus is fully annotated, so a predicted span overlapping nothing is a false positive.
• F1 here combines an entity-level recall with a span-level precision. It is an indicative harmonic mean, not a strict span-exact F1.
• Classes are grouped into deterministic (dob · govid · iban · card · phone · email — decidable from the characters alone) and free-form (person · org · place — open-vocabulary, needs a gazetteer or a model). These two groups behave so differently that merging them into one headline number would be misleading, so we never do.

Deterministic classes

dob · govid · iban · card · phone · email

system	recall (any)	recall (typed)	precision (any)	precision (typed)	F1 (any)	non-Latin recall
CloakAPI A	99.7%	97.8%	100.0%	98.4%	99.8	99.3%
CloakAPI B	99.7%	97.8%	100.0%	98.4%	99.8	99.3%
Presidio	90.1%	87.8%	99.8%	82.0%	94.7	87.9%
OpenAI Privacy Filter	93.0%	63.5%¹	99.4%	91.2%¹	96.1	94.5%

¹ Privacy Filter’s account_number label covers IBAN, card and gov-ID alike, so it cannot be mapped to one class. Its typed figures are structurally understated and should not be read as a comparison.

Free-form classes

person · org · place

system	recall (any)	recall (typed)	precision (any)	precision (typed)	F1 (any)	non-Latin recall
CloakAPI A	61.7%	51.7%	89.0%	79.1%	72.9	58.4%
CloakAPI B	93.7%	88.3%	80.6%	74.4%	86.7	92.9%
Presidio	48.8%	36.7%	64.2%	43.3%	55.5	26.3%
OpenAI Privacy Filter	67.5%	52.8%	90.5%	68.0%	77.3	63.9%

Per class

Recall (any) — did the system flag the entity at all?

system	person	org	place	dob	govid	iban	card	phone	email
CloakAPI A	52.2%	42.5%	95.0%	99.2%	100%	100%	100%	100%	100%
CloakAPI B	95.3%	81.7%	99.4%	99.2%	100%	100%	100%	100%	100%
Presidio	57.5%	34.2%²	42.5%	78.3%	85.7%	96.7%	100%	90.6%	100%
OpenAI PF	93.4%	20.0%³	51.3%	85.0%	100%	100%	100%	100%	89.0%

Precision (any) — of what it flagged, how much was actually an entity?

system	person	org	place	dob	govid	iban	card	phone	email
CloakAPI A	84.8%	100%	90.1%	100%	100%	100%	100%	100%	100%
CloakAPI B	88.9%	70.3%	74.5%	100%	100%	100%	100%	100%	100%
Presidio	61.8%	n/a²	70.9%	99.4%	100%	100%	100%	100%	100%
OpenAI PF	89.4%	n/a³	98.2%	98.5%	n/a¹	n/a¹	n/a¹	100%	100%

² Presidio’s own shipped default configuration excludes ORGANIZATION — its conf/default.yaml lists it among the labels to ignore, with the upstream comment “Has many false positives”, and get_supported_entities() confirms it is absent. Its org row is therefore outside its default entity set; the 34.2% is organisations incidentally caught by its PERSON/LOCATION labels. This is not a defeat of Presidio.

³ OpenAI Privacy Filter’s label taxonomy contains no organisation label at all. Same caveat: the org column is not a like-for-like comparison, and is published to document our own coverage rather than to score a point.

Per script, all classes, any-recall:

system	Latin	non-Latin
CloakAPI A	84.1%	80.8%
CloakAPI B	97.4%	96.4%
Presidio	80.1%	60.0%
OpenAI PF	82.3%	80.6%

Where CloakAPI loses

Publishing a benchmark of your own product is only worth anything if the losses are in it. These are ours, on this corpus.

1. Overall precision — OpenAI Privacy Filter beats us: 95.8% vs 90.4%. Our shipped configuration produced 139 false positives to Privacy Filter’s 96, even though it emitted fewer spans in total (1454 vs 2279).
2. Free-form precision — Privacy Filter 90.5% vs CloakAPI 80.6%. We buy free-form recall (93.7% vs 67.5%) at the cost of precision. For a redaction tool that is a deliberate posture — over-redaction is recoverable, missed PII is not — but on the precision axis it is a loss, and we are not going to dress it up as anything else.
3. place precision — Privacy Filter 98.2% vs CloakAPI 74.5%. It is far more conservative about places (56 place spans to our 235). It pays for that with 51.3% place recall against our 99.4%, but on precision alone it wins clearly.
4. person precision — Privacy Filter 89.4% vs CloakAPI 88.9%, marginally ahead of us. (Recall runs the other way: 95.3% ours, 93.4% theirs.)
5. Speed — Presidio is faster than either of our configurations: 11.9 ms/text against our 15.7 ms/text (engine only) and 342 ms/text (with the on-device model). Privacy Filter is the slowest at 1014 ms/text. All four were measured on the same shared box in the same session, so treat these as an order-of-magnitude comparison rather than a latency benchmark.
6. Classification, not detection, on two classes: we detect 100% of gov-IDs and phones but assign a different class to 2 of 28 gov-IDs and 12 of 160 phones (typed recall 92.9% and 92.5%). And 12 of 118 card-typed spans do not overlap a gold card entity (typed precision 89.8%).
7. email is a tie, not a win — CloakAPI and Presidio both reach 100%; only Privacy Filter drops, to 89.0%.
8. Without the download, free-form recall is 61.7%. Config A — the zero-download path — gets 58.4% free-form recall on non-Latin scripts. The strong free-form numbers belong to config B and require fetching a 135 MB model onto the device.

The dominant false-positive mode in our shipped configuration is the on-device model firing on non-English form-label words — the corpus’s own “Name:”, “Phone:”, “City:” label words in Czech, Korean, Tamil, Hebrew and others get classified as a person, organisation or place. That accounts for the bulk of our 139 false positives. It is recorded as input to a future engine revision; it was not changed for this benchmark, because the engine is pinned at one build across every shipped product and re-cutting it would invalidate that verification.

Where CloakAPI does better on this corpus

Deterministic classes (99.7% any-recall at 100% any-precision, against Presidio’s 90.1% and Privacy Filter’s 93.0%); non-Latin recall across all classes (96.4%, against 80.6% and 60.0%); free-form recall (93.7%, against 67.5% and 48.8%); and dob (99.2% against 85.0% and 78.3%), where native-script date constructs break both baselines.

Known biases and limits of this corpus

These are properties of the corpus, not of any system measured on it. They apply identically to all four configurations, but they shape the numbers.

• Form-heavy register. Seven of eight templates per language are labelled admin forms. Model-based detectors mis-fire on the non-English label words far more than they would on prose, which depresses free-form precision for every model-driven system here — including ours.
• One entity value per class per language. Places and organisations reuse one verified label across a language’s texts, so knowing or missing that one label moves three to four gold entities at once. Real confidence intervals are wider than the raw counts suggest.
• Thin classes. govid has 28 gold entities and iban 60. One govid miss moves that row by 3.6 points. Small differences in these rows are noise.
• Organisations carry no label cue — they appear as a bare letterhead line, because hand-translating an “Employer:” label into 40 languages would have put unverifiable translation quality into the gold set. Org is therefore the hardest, most cue-free class in the corpus, for every system.
• Synthetic and clean. No OCR noise, no typos, no mid-sentence code-switching, no deliberate obfuscation. Real traffic is messier, and every system here should be expected to do worse on it.
• Equal weight per language, which is what makes the non-Latin split meaningful, but does not reflect any real traffic mix. 23 of the 40 languages are Latin-script; within the non-Latin half, Cyrillic is the best-represented script.
• One execution-provider difference on our side: in the browser the on-device model runs on onnxruntime-web (WASM); the offline runner used for these numbers runs it on onnxruntime-node (CPU). Same ONNX file, same int8 weights, same tokenizer, same thresholds — different execution provider. The WebAssembly PII engine itself is byte-identical, and its sha-256 is recorded in every results file.

Reproducing it

The corpus generator, the offline runner, the shared scorer and both baseline recipes are kept together as one tree, with the raw predictions and scored results for every system. The runner executes the same WebAssembly engine the extension, portal and desktop apps ship — in plain Node, with no CloakAPI account, no gateway and no network access — and prints the engine’s ruleset version and sha-256 so any published figure can be tied to an exact build.

cd corpus  && node generate.mjs      # deterministic; prints the corpus sha-256
cd ../runner && npm i
node run.mjs                          # config A — engine only
node run.mjs --ner                    # config B — + gazetteer + on-device model
node score.mjs ../corpus/corpus.sample.json <predictions.json>

Raw artifacts for this run — per-system predictions and scored results, the corpus, its manifest (corpusVersion 1.0, sha-256 696c4e68…417c99c1), the Wikidata verification transcripts, and the frozen baseline environments — are held in the benchmark tree under goal-run/benchmark/ (results/, corpus/, baselines/), with the full numbers in goal-run/benchmark/RESULTS.md. The tree is prepared for public release alongside the corpus; that release is not yet published.

Numbers are always cited as (corpus version, engine version). This page is corpus v1.0, engine ruleset 1.4.29. Every engine release re-runs all systems on the frozen corpus; any corpus change bumps the corpus version and requires re-running every system, because predictions are corpus-specific.