Guide 16: Methods and Materials

The exact prompts, output schema, scoring, datasets, and reproduction steps behind the intake-classification evaluation. The scoring philosophy is in Guide 3, and the LIST taxonomy in Guide 14.

Conditions and models

Four prompt configurations, identical except for the reference material in context. A is the control. C adds the full 1,124-code LIST taxonomy. D adds the audience taxonomy. G is the full kit, C plus D plus a FIPS reference.

Gemini ran with all four adjustable safety thresholds set to allow everything, which did not resolve its empty-response problem:

safetySettings:
  - category: HARM_CATEGORY_HARASSMENT
    threshold: BLOCK_NONE
  - category: HARM_CATEGORY_HATE_SPEECH
    threshold: BLOCK_NONE
  - category: HARM_CATEGORY_SEXUALLY_EXPLICIT
    threshold: BLOCK_NONE
  - category: HARM_CATEGORY_DANGEROUS_CONTENT
    threshold: BLOCK_NONE

promptfoo options: maxConcurrency: 2, delay: 1000ms, needed to stay under the GPT-4o tier-1 rate limit on condition G.

The base prompt (condition A)

Conditions C, D, and G prepend reference material to this same prompt without changing it:

You are a legal-help intake classifier. Read the user's input from a legal aid intake channel, then return a single JSON object classifying their legal situation.
Required JSON fields:
- list_codes: array of LIST taxonomy codes (format like "HO-02-03-02-01"). One or more if the user describes multiple legal issues. Do NOT invent codes.
- top_parents: array of LIST top-level category names that match those codes (e.g. "Housing", "Family", "Public Benefits").
- state: full US state name where the legal issue is located, or "" if unclear.
- fips_code: numeric FIPS code as a string. Use county-level FIPS where the user mentions a county or city. State-only FIPS if no county detail. "" if unclear.
- audience_flags: array of audience-served codes (from a closed vocabulary). Apply a flag only when there is explicit or strongly implied evidence in the query. May be empty.
- emergency: true if the user faces (a) an imminent court date, (b) physical danger, (c) a major immigration consequence such as removal or detention, (d) a mental health or medical crisis, (e) a child at risk, or (f) a lockout / set-out from housing. Otherwise false.
- reasoning: one short sentence explaining the key classification.
Return ONLY the JSON object. No preamble. No markdown fences. No commentary after the JSON.
User input (channel: {{input_channel}}):
{{user_query}}

Reference material added per condition

C, full LIST taxonomy. All 1,124 codes, one per line, formatted CODE | Term Name (under: Top Parent), prepended to the base prompt:

=== LIST TAXONOMY - FULL CODES (REFERENCE) ===
BE-00-00-00-00 | Public Benefits (under: Public Benefits)
BE-01-02-00-00 | Social Security Disability Insurance (SSDI) (under: Public Benefits)
BE-01-03-00-00 | Supplemental Security Income (SSI) (under: Public Benefits)
... [1,124 codes, by 20 top parents] ...
WO-11-00-00-00 | Suing an employer for a tort (under: Work and Employment Law)

The full list is data/valid_list_codes.txt, and the taxonomy with definitions is on taxonomy.legal and in Guide 14.

D, audience taxonomy. All 22 audience tags, each with a definition, signals to look for, typical phrases, and disambiguation notes. One example entry, for the domestic-violence tag:

## domestic-violence - Domestic Violence
Definition: Has experienced or is currently experiencing intimate partner violence, family violence, or stalking by a current or former partner or household member.
Signals to look for: Mentions of being hit, threatened, choked, raped, or scared of partner; partner has gun; restraining/protection order references; DV shelter
Typical phrases: "he hits me"; "I am scared of him"; "I have a PPO"; "I am in a shelter"
Notes: Distinct from elder abuse and child abuse. Co-occurs heavily with parents-with-minors.

The full vocabulary is data/audience_codes.txt, available by request with the harness.

G, full kit. C plus D plus a FIPS reference. The standalone FIPS lookup table was dropped after it did not help and slightly hurt; county FIPS is still scored in every condition.

Output schema

Every condition asks for one JSON object with seven fields:

{
  "list_codes": ["FA-04-09-00-00", "FA-12-00-00-00"],
  "top_parents": ["Family"],
  "state": "Tennessee",
  "fips_code": "47157",
  "audience_flags": ["citizens-permanent-residents", "lgbtq", "parents-with-minors"],
  "emergency": false,
  "reasoning": "Brief free-text explanation of the classification."
}

The 20 canonical top parents: Public Benefits, Small Business and IP, Courts and Lawyers, Money Debt and Consumer Issues, Crime and Prisons, Disaster Relief, Education, Environmental Justice, Estates Wills and Guardianships, Family, Government Services, Health, Housing, Native American Issues and Tribal Law, Immigration, Civil and Human Rights, Accidents and Torts, Traffic and Cars, Veterans and Military, Work and Employment Law.

The high-stakes audience subset, scored by audience_high_stakes_recall: domestic-violence, youth, experiencing-homelessness, incarcerated, immigrants-refugees, native-american-tribal, veterans-military. These are the flags where a miss means wrong routing, missed safety planning, or missed eligibility.

Scoring

All scoring is deterministic Python, no LLM judge. The 14 metrics:

Audience normalization lowercases, swaps underscores for hyphens, and strips non-alphanumerics, so LGBTQ+, lgbtq, and LGBTQ_plus compare as one tag. FIPS normalization strips trailing .0 and zero-pads to five digits. The scoring code is scripts/regrade.py.

Test cases

60 stratified scenarios drawn from a 180-case synthetic set, stratified by difficulty, input channel, and the is_legal_issue flag. 53 are real legal scenarios with gold codes, audience flags, and jurisdiction; 7 are deliberately non-legal, like a barking-dog complaint or retirement planning, where the right answer is an empty list. The full set is data/test_cases_stage2.csv.

Datasets on JusticeBench

• Intake classification inputs
• FIPS jurisdiction list
• LIST taxonomy
• Audience codes

Run results

Eval IDs: Stage 1 eval-rlw-2026-05-19, Stage 2 first pass eval-CCB-2026-05-19, and a Stage 2 final pass with the Gemini safety adjustment and FIPS normalization. Per-row scores are results/stage2_per_row_scores_v2.csv and the per-condition summary is results/stage2_summary_by_condition_v2.csv. These and the promptfoo runs are available by request, and can be hosted behind this gate.

Reproduce it

The harness is available by request, shared under an open license once you have access.

jkb-eval-classifier/
├── data/
│   ├── test_cases_stage2.csv       # 60 stratified scenarios with gold
│   ├── valid_list_codes.txt        # 1,124 LIST codes (one per line)
│   └── audience_codes.txt          # audience taxonomy
├── prompts/
│   ├── A_control.txt
│   ├── C_full_taxonomy.txt
│   ├── D_audience.txt
│   └── G_full_kit.txt
├── scripts/
│   ├── grade.py                    # promptfoo deterministic graders (live)
│   ├── regrade.py                  # post-hoc rescoring with expanded metrics
│   └── build_stage2_sample.py      # stratified sample construction
├── promptfooconfig.stage2.yaml
└── results/
    ├── stage2_results.json
    ├── stage2_per_row_scores_v2.csv
    └── stage2_summary_by_condition_v2.csv

git clone <repo url>
cd jkb-eval-classifier
npm install -g promptfoo
# Set API keys in .env: ANTHROPIC_API_KEY, GOOGLE_API_KEY, OPENAI_API_KEY
promptfoo eval -c promptfooconfig.stage2.yaml
python3 scripts/regrade.py results/stage2_results.json --out-dir results/

Request access to the evaluation harness →