All benchmarks
Workforce and operations

Courses, manuals and training materials for staff, inspectors and the public, built from the rules they teach.

Deliverable
A training course or manual with its sources

Training and course design · overall

Deliverable quality score

  • Justinian3.2A96.3%
  • ChatGPTAstra 6.2A92%
  • ClaudeFable 5.2A92.3%
  • Grok4.7F35%
  • Gemini4 CarbonB80.3%

Checklist

What the judges checked

Written before any system ran.

  1. 01A .pptx deck with 12 to 16 slides and speaker notes is delivered
  2. 02A one-page Word facilitator guide with timing for 45 minutes is delivered
  3. 03A Word quiz with 10 questions and an answer key is delivered
  4. 04All procedure details are accurate (1-day acknowledgment, 3-day production or estimate, 10-day cap, fee rules incl. $5 waiver, redact-not-withhold, AG appeal within 30 days)
  5. 05Quiz questions test the procedure (not trivia) and the key is correct
  6. 06Materials are visually clean and usable as delivered

Agencies that use it

  • Environment, Water & Natural Resources
  • Health, Human Services & Veterans
  • Public Safety, Corrections & Emergency Management
  • Education & Workforce
  • Agriculture & Rural Services

Example tasks

What the systems were asked to do

Real tasks from this benchmark. Every system got the same prompt and data.

Course design

Training and course design

Build a 45-minute new-employee training on our public records request procedure: a slide deck (12 to 16 slides) with speaker notes, a one-page facilitator…

Full task
Build a 45-minute new-employee training on our public records request procedure: a slide deck (12 to 16 slides) with speaker notes, a one-page facilitator guide (Word), and a 10-question quiz with an answer key (Word).
Procedure: Requests may be made orally or in writing; only written requests start the clock. Acknowledge within 1 business day. Produce records or a written estimate within 3 business days; if more time is needed, give a written date no later than 10 business days. Fees: no charge for the first 50 pages; $0.10 per page after that; staff time over 2 hours at $15 per hour; waive fees under $5. Exempt records: personnel evaluations, active criminal investigations, attorney-client communications, social security numbers (redact, do not withhold the whole record). Denials must cite the exemption in writing and state the right to appeal to the Attorney General within 30 days. Log every request in the tracking system the same day.

Task types

What this benchmark covers

Course design2 tasks
Courses and lesson plans built from the governing rules.
Assessments1 task
Quizzes and checks that test what the rules require.
Public education2 tasks
Plain-language materials for the public and regulated parties.

Method

How we test

  1. 01

    Real agency deliverables

    One task per deliverable use case: trackers, dashboards, permit systems, forms, decks, workbooks, letters and one-pagers. Each carries its own rules and data, with problems planted where a careful reviewer would catch them.

  2. 02

    Every system can make files

    ChatGPT Astra 6.2, Claude Fable 5.2, Grok 4.7 and Gemini 4 Carbon run on their makers' own services with web search, high reasoning effort and a code tool whose files are what the user receives. When their code crashed or sent no files, they got up to two follow-ups. Justinian ran as shipped, with none.

  3. 03

    Graded as delivered

    Every file is rendered as the requester would open it and every app is opened in a browser. Code or instructions the user would still have to run, deploy or schedule do not count. Bill numbers and rule sections are checked against the official record, and every regulatory call is checked against the statute.

  4. 04

    Three blind judges

    Judges from OpenAI, Anthropic and Google, none of them a competing model, score each deliverable as the user would see it and rank all five systems with names masked. Every system is graded with the same checklist and scoring guide.

Briefing

Test Justinian on your agency's own work.

Request a briefing →

This page’s interactive features did not load. Reloading usually fixes it. You can email sales@vulcan.ai for demos or support@vulcan.ai for help.