// Name detection and synthetic tests
Results, with the limits in view.
A new test build improves masking and restoration. The name model remains unchanged after the latest training round. Here are the new development measurements, unmet targets and historical results, with their limits.
DMZ · new test build · 05.10.2026
More identifiers masked in full.
A new private test build improves masking of email addresses, credentials and supported numbered street addresses without postcodes. Restoration preserves the original text, including accented characters, and pending scans are handled more reliably. A draft edited while waiting is not sent automatically.
These changes are in a test build, not a general release. The name model is unchanged and the 99% detection target remains unproven.
| Probe | Before | After |
|---|---|---|
| Email variants | 18 / 39 | 39 / 39 |
| AWS presigned URL cases | 1 / 30 | 30 / 30 |
| GitHub tokens | 1 / 15 | 15 / 15 |
| Credential assignments | 0 / 25 | 21 / 25 |
| Temporary AWS credential components | 7 / 23 | 21 / 23 |
These are selected, reused development probes, not representative samples or independent estimates of real-world performance. Masking part of an identifier does not count as complete masking. Whole-text masking can protect a value without correctly detecting its type and boundaries. These probes must not be pooled into an overall accuracy rate.
Restoration and submission handling checked.
Every evaluated excerpt restored exactly across 20 synthetic corpora, with both basic masking and the trained detector. Sixteen submission and failure-handling workflows were exercised in the browser. Exact restoration in these tests does not mean every identifier was detected.
Names, addresses outside the supported formats, and some short or unusual credentials are still missed. Benign references can also be masked incorrectly. Independent annotation and testing of the new build in everyday use are the next steps.
Name detection · development status · 05.10.2026
The name model remains unchanged.
The latest training round did not produce a model that passed comparison with the current model. Better recall on synthetic legal and financial text came with lower recall on general text. The existing model remains in the separate DMZ test build. There is no general release.
| Metric | General text · 5,582 excerpts | Synthetic legal/financial text · 256 documents |
|---|---|---|
| Excerpts with every name correct | 98.57% | 97.66% |
| Name recall | 97.01% | 98.75% |
| Detection precision | 95.55% | 99.37% |
| Missed name spans | 45 | 6 |
| False detections | 68 | 3 |
The target is at least 99% exact entity-set accuracy and 99% recall in each primary text category. It has not been reached. Independent validation is still pending.
An excerpt is correct only when every annotated entity has the right type and exact span, with no extra detections. Recall measures the share of annotated entities found. Precision is a different metric: the share of detections that are correct. This table evaluates names only; the acceptance target covers all entity types.
These are development results after repeated model and parameter selection, not independent final validation. The synthetic documents reuse templates and annotation review is pending. These figures do not establish everyday performance or the accuracy of DMZ as a whole.
Validation progress
A packet of 142 excerpts from 88 documents awaits independent annotation and review. It has not been run through the model. Annotation review and adjudication remain pending; no independent final score is available. Document separation and representative sampling must be established before claiming everyday performance.
An earlier Chromium research evaluation reproduced the model’s precision and recall. The new DMZ test build has also been exercised with synthetic masking, restoration and failure-handling cases. This does not establish performance across all sending paths or real work documents.
Earlier synthetic tests · archived measurements
The tables below preserve earlier measurements on 42, 18 and 29 synthetic cases. They describe the builds tested then, not the newly trained detector. The Presidio comparison applies only to that earlier synthetic set.
SEPARATE CHALLENGE SET
2 known misses in a synthetic test.
These 18 cases were written after the initial detector rules. The original run exposed 8 annotated identifiers. The rules were then tuned to these cases; the table preserves the result from that round on the same set. It is neither an independent test nor a representative leak rate.
Recorded exact-span recall: names 80%, addresses 100%, secrets 66.7%, phones 100%, emails 100%. Each type has only a handful of examples.
EARLIER EXPLORATORY PROBE
13 identifiers in allowed text on 29 cases.
These cases were written after the fixes to probe other formats and false positives. The detector was not tuned to them. This small, targeted set is not a random sample or a real-world rate estimate. One benign price line was blocked as an address.
dmz prototype by type.
A true positive requires the right type and exact text span. FP means a false detection; FN means a missed identifier.
TP = true positive, FP = false positive, FN = missed entity. The dictionary test has one preconfigured synthetic term, Verkefni Gervi. The single address hits a narrow shape rule and blocks the prompt.
PROTOTYPE OUTBOUND BOUNDARY
What would still get through?
On this selected corpus, 0 annotated identifiers remain in allowed outbound text. 2 unfamiliar full names and 1 address block their prompts for review; 10 prompts are blocked overall. This is not evidence of zero leaks on real prompts.
Counted per annotated identifier: masked, blocked with the whole prompt, or still present in allowed text. Nothing was sent to a model. Unannotated context is not assessed, and the corpus is not a representative leak-rate sample.
Comparison: plain regex.
The same 42 cases, without date, checksum, mod-97 or Luhn validation and without a name lexicon. Patterns may overlap; each wrong type counts as a false positive.
Comparison: default Presidio.
Presidio Analyzer 2.2.364 with its default English en_core_web_lg 3.8.0 model. No Icelandic recognizer or custom configuration was added. The same exact spans and types are scored.
How we measured.
Every case has annotated identifiers. The Go detector returns an entity type and UTF-8 byte offsets. The scorer compares exact span and type; partial overlaps do not count.
Test kennitölur use century digit 8. Emails use reserved example domains, cards use test numbers and no customer data is included.