How Can We Automatically Mask PII in Documents?
Legal help organizations sit on confidential records that could improve services, research, and safer tools, once the identifying information is removed. This guide is what the cohort's expert survey and document work produced: what counts as PII in legal records, how to judge whether an automatic masking tool actually works, and how to test one without touching a real client's file.
Common Legal Help AI. Working draft for comment. Margaret Hagan, Stanford Legal Design Lab.
Legal aid groups, courts, and clinics handle large volumes of confidential data: case notes, filings, intake records, and communications. That data could improve services, support research, inform policy, and train safer tools, and none of that is allowed until the identifying information is removed. Automatic PII masking is the task of doing that removal reliably enough to trust. The cohort worked on the three things a team needs before it can rely on a masker: a shared definition of what counts as PII, a rubric for judging a tool's output, and a document set to test against.
What the task is
A PII masker scans confidential records, including text fields, transcripts, and uploaded documents, finds the personally identifying information, and either removes it or replaces it. Redaction takes the information out. Replacement swaps the real value for a synthetic identifier, a fictional and non-reversible stand-in that stays internally consistent, so a client named Martha Pizzoli becomes Client A07 everywhere she appears across a record set. Replacement keeps the structure and meaning of the data, which is what lets a de-identified record stay useful for research or training, while making sure no real person can be re-identified. A good tool also produces a masking manifest, a machine-readable record of what it found and how it transformed each item, and an audit log of who ran it and under which policy.
What counts as PII, the part that is not software
The hard part is not the detection software. It is agreeing on what to mask. The Lab surveyed subject-matter experts at legal help teams on common legal document fields, asking each to mark a field as clear PII, possible PII, or not PII. The answers sorted into three tiers.
High risk, mask by default: full SSN, bank and financial account numbers, driver's license or state ID, passport number, phone, email, date of birth, signature, home, mailing, and property address, insurance policy number, and criminal history details.
Medium risk, mask depending on context: the last three digits of an SSN or ID, case or court number, initials, age, income and assets, medical condition, pregnancy status, employer and school or daycare names, VIN, license plate, and professional license numbers, place of birth, immigration identifiers such as an A-number, and address components like a ZIP code or census tract.
Low risk, usually leave alone: hearing date and time, document date, rent amount, generic financial totals, judge name, insurance company name, and laws, citations, and legal labels.
One distinction runs underneath the tiers. Many teams separate identifiers, which point to a person, from sensitive information such as medical, pregnancy, and income details. A team may choose to treat sensitive fields as maskable even when they are not strictly identifiers, and that is a policy decision to make on purpose rather than leave to the tool.
How to tell whether a masker is any good
A masking tool can look like it worked and still leak, so the cohort built a rubric that scores the output. It starts with hard fails that override everything else. A tool fails outright if a high or definite PII item is still visible, if the PII can still be found by searching or copy-pasting because the text layer was never cleaned, or if the masking is reversible because it is an annotation or a layer that can be peeled off. A black bar over text that is still selectable underneath is the classic version of this failure.
Past the hard fails, five questions get scored on a one-to-three scale. Does it catch the PII that is there, including every repeat. Does it mask the right things and leave the low-risk content alone. Is the masking real, meaning the information is gone rather than hidden, and it survives export to PDF, image, or plain text. Does it avoid overmasking, so it does not black out whole paragraphs over one identifier and the legal meaning stays readable. And is it consistent and reviewable, catching every instance across headers, footers, and tables, and showing what it masked so a person can verify and fix it. A tool with no hard fails and mostly threes is good, mostly twos is usable with a cleanup pass, and a hard fail or several ones means it is not ready.
The safety asymmetry is the thing to hold onto. A missed high-risk item is far worse than an over-masked low-risk one, because the false negative is the one that exposes a real person.
How to test it without real client data
You do not test a masker on real client files, because a tool that fails would expose the exact information you are trying to protect. The cohort assembled a legal document set for this instead. It spans the document types teams actually handle, including eviction notices, summonses, complaints, court orders, motions, notices, citations, and hearing notices, across states, and each document is tagged with its type, jurisdiction, and legal issue, with the identifying fields marked: the party names, addresses, phone numbers, case numbers, and amounts. Some are blank forms and some are filled with synthetic values, and the hard cases are in on purpose: all-caps notices, scanned pleadings, headers and footers, signatures, stamps, and tables.
A team runs its masker over the set and scores the output against the rubric, measuring recall by entity type, with the high-risk entities watched most closely, precision so it can see overmasking, and whether each item was given the right type. Because the documents and their identifying fields are known in advance, the test is repeatable and safe to run as often as the tool changes.
What automatic masking still does not solve
Masking gets a team most of the way and not the whole way. The medium tier is context-dependent by definition, so a tool will sometimes guess wrong, which makes a human review pass part of the workflow rather than an optional extra, especially before any data leaves the organization. Handwriting, stamps, and low-quality scans are harder than clean digital text, and non-English names raise the miss rate. A masker tuned for one document type will miss the quirks of another, so the test set has to grow to cover the documents a team actually handles. The reliable shape is a tool that does the bulk detection and a person who reviews the manifest and the flagged low-confidence cases before anything is shared.