Courses, manuals and training materials for staff, inspectors and the public, built from the rules they teach.
- Deliverable
- A training course or manual with its sources
Training and course design · overall
Deliverable quality score
- Justinian3.2A96.3%
- ChatGPTAstra 6.2A92%
- ClaudeFable 5.2A92.3%
- Grok4.7F35%
- Gemini4 CarbonB80.3%
Checklist
What the judges checked
Written before any system ran.
- 01A .pptx deck with 12 to 16 slides and speaker notes is delivered
- 02A one-page Word facilitator guide with timing for 45 minutes is delivered
- 03A Word quiz with 10 questions and an answer key is delivered
- 04All procedure details are accurate (1-day acknowledgment, 3-day production or estimate, 10-day cap, fee rules incl. $5 waiver, redact-not-withhold, AG appeal within 30 days)
- 05Quiz questions test the procedure (not trivia) and the key is correct
- 06Materials are visually clean and usable as delivered
Agencies that use it
- Environment, Water & Natural Resources
- Health, Human Services & Veterans
- Public Safety, Corrections & Emergency Management
- Education & Workforce
- Agriculture & Rural Services
Example tasks
What the systems were asked to do
Real tasks from this benchmark. Every system got the same prompt and data.
Course design
Training and course design
Build a 45-minute new-employee training on our public records request procedure: a slide deck (12 to 16 slides) with speaker notes, a one-page facilitator…
Full task
Build a 45-minute new-employee training on our public records request procedure: a slide deck (12 to 16 slides) with speaker notes, a one-page facilitator guide (Word), and a 10-question quiz with an answer key (Word). Procedure: Requests may be made orally or in writing; only written requests start the clock. Acknowledge within 1 business day. Produce records or a written estimate within 3 business days; if more time is needed, give a written date no later than 10 business days. Fees: no charge for the first 50 pages; $0.10 per page after that; staff time over 2 hours at $15 per hour; waive fees under $5. Exempt records: personnel evaluations, active criminal investigations, attorney-client communications, social security numbers (redact, do not withhold the whole record). Denials must cite the exemption in writing and state the right to appeal to the Attorney General within 30 days. Log every request in the tracking system the same day.
Task types
What this benchmark covers
- Course design2 tasks
- Courses and lesson plans built from the governing rules.
- Assessments1 task
- Quizzes and checks that test what the rules require.
- Public education2 tasks
- Plain-language materials for the public and regulated parties.
Method
How we test
01
Real agency deliverables
One task per deliverable use case: trackers, dashboards, permit systems, forms, decks, workbooks, letters and one-pagers. Each carries its own rules and data, with problems planted where a careful reviewer would catch them.
02
Every system can make files
ChatGPT Astra 6.2, Claude Fable 5.2, Grok 4.7 and Gemini 4 Carbon run on their makers' own services with web search, high reasoning effort and a code tool whose files are what the user receives. When their code crashed or sent no files, they got up to two follow-ups. Justinian ran as shipped, with none.
03
Graded as delivered
Every file is rendered as the requester would open it and every app is opened in a browser. Code or instructions the user would still have to run, deploy or schedule do not count. Bill numbers and rule sections are checked against the official record, and every regulatory call is checked against the statute.
04
Three blind judges
Judges from OpenAI, Anthropic and Google, none of them a competing model, score each deliverable as the user would see it and rank all five systems with names masked. Every system is graded with the same checklist and scoring guide.
More in Workforce and operations
All benchmarksBriefing
