Can AI Tag an Intake the Way an Intake Worker Does?
A 60-case eval across Claude, Gemini, and GPT-4o on reading a person's own words and tagging the legal issue, who they are, where they are, and whether it is urgent.
Common Legal Help AI. Working draft for comment. Margaret Hagan, Stanford Legal Design Lab.
Intake workers do a specific job every day: take a person's description of their problem and tag it. What kind of legal issue is this. Who is the person, a tenant, a domestic violence survivor, a veteran, someone who is unhoused. What state and county are they in. Is this an emergency. This round tested whether AI can do that job, and what it takes to make it work.
We ran 60 stratified scenarios through three frontier models, Claude Sonnet 4.6, Gemini 2.5 Pro, and GPT-4o. The cases came from a 180-case set of synthetic intake inputs: 53 are real legal problems with known correct tags, and 7 are deliberately not legal problems, like a financial-planning question or a barking-dog complaint, where the right answer is an empty list. Each model ran under four prompt setups: a bare control, the control plus the full LIST taxonomy, the control plus our audience categories, and the full kit with both plus a FIPS reference. Scoring was deterministic Python against gold values, no LLM judge, on 14 metrics that track under-classification separately from over-classification, since missing a flag is the more serious harm. The whole round cost about $45.
The finding: put your category lists in the prompt
The result is simple. Giving the model the actual lists your team uses is what takes it from useless to useful. The biggest lift is on the flag that matters most for routing: whether the model recognizes that a person is a domestic violence survivor, unhoused, incarcerated, an immigrant, Native or tribal, or a veteran. Without the audience list in front of it, the model catches about one in ten of these. With the list, eight or nine in ten on the two strongest models.
The legal-issue tagging moves the same way. Going from the bare control to the full LIST taxonomy, recall on legal cases rose on every model, with Claude reaching the highest level.
Jurisdiction held up well without extra help. Claude and GPT-4o returned the correct county FIPS code about nine times in ten, inferring the county from a city, zip, or county name in the person's description. Loading a FIPS lookup table into the prompt did not help and slightly hurt, so the practical move is to ask for the county and give the model enough location detail in the input, then do any code lookup on your own side.
Where it breaks
Two failure points are worth carrying forward.
Gemini 2.5 Pro returned nothing at all on roughly half of legal scenarios. Its legal-issue recognition sat at 0.38 to 0.43 across conditions, against 1.00 for Claude and near 1.00 for GPT-4o. The first guess was a safety filter on sensitive content like domestic violence or criminal records, so we reran with all four adjustable safety thresholds set to allow everything. The numbers barely moved. The failing cases returned empty strings, not refusals, so the model is silently producing no output rather than output we cannot parse. That makes "test across vendors before you deploy" a quantified finding rather than a slogan: the same prompt that runs cleanly on one model returns nothing on another half the time.
GPT-4o could not finish the full-kit condition. That prompt runs to about 15,000 input tokens, and the tier-1 rate limit is 30,000 tokens per minute, so the runs exhausted their retries. It is solvable with a higher tier or a higher-limit model, and the cross-vendor story was already clear from the audience-list condition, so we left it for the next stage.
How to build it, for legal help teams
If you take one thing from this round: give the model the actual lists your team uses, and put the whole list in the instructions it sees with every request. Term names and the parent hierarchy are enough, definitions help but are not required. Five things follow from that.
Force the model to pick from your list rather than generate freely, and specify the output as codes from the attached list and no others. Run the classifier as one component in a larger system, since on its own it will not reliably catch "this isn't really a legal issue," so you need a downstream check for that. Run emergency detection as its own dedicated check rather than as one field among many, because emergency recall fell from 1.00 to between 0.86 and 0.93 once the full taxonomy was in context. Expect the model to return more codes than the gold answer holds, and design your routing for the top several rather than a single top answer.
This works even if your lists are not ours. The lift comes from giving the model a structured list to pick from, not from LIST specifically. Put your own list of legal issues in the prompt with its parent categories shown, and you get the same mechanism. A smaller list than LIST, which has 1,124 codes, may even score better, since the model has fewer ways to over-classify. The one thing worth borrowing from LIST directly is the code format, which is machine-readable and increasingly common, and which lets your data flow into a multi-state system later if you build a crosswalk. The same holds for audience categories: use your own, as long as the list is complete and shown to the model, and make sure your intake is actually capturing the high-stakes flags, because the model cannot route on a signal your system never collects.
For funders and state strategists
For this task, AI is worth investing in, with one condition: the lift depends on the taxonomies and category lists being in place and maintained. The model is the cheap part. The lists, the audience vocabulary, the jurisdiction codes, and the work of keeping them current are the infrastructure that makes the model useful, and they are shared across organizations rather than rebuilt by each one. The Gemini result is also a direct argument against betting a state's intake on a single vendor: the same setup that performed on two models failed quietly on the third, so portability across models is a requirement, not a nicety.
What this round did not settle
The numbers are preliminary. N is 60, the inputs are synthetic, and the scoring is deterministic, so it measures whether the tags match gold, not the softer questions of tone or judgment that a human reviewer would catch. The Gemini under-classification is observed but not diagnosed, and the GPT-4o full-kit condition is missing. Stage 3 needs to resolve the Gemini behavior, finish the full kit on a higher-limit model, widen the case set, and test on the channels intake actually arrives through, including voice transcripts and text messages.
See the methods and materials: prompts, schema, scoring, datasets, and reproduction steps →