All benchmarks
Workforce and operations

Reports, minutes, briefings, talking points, web content and letters, accurate to the record.

Deliverable
A finished document ready for review

Reports, briefings and correspondence · overall

Deliverable quality score

  • Justinian3.2A92%
  • ChatGPTAstra 6.2B78%
  • ClaudeFable 5.2A-87.3%
  • Grok4.7C+70%
  • Gemini4 CarbonC+67.7%

Checklist

What the judges checked

Written before any system ran.

  1. 01A Word report is delivered with an executive summary, a comparison with the second quarter, at least one table and one chart, explanations and recommendations
  2. 02Call volume is correct: 54,857 calls in Q3 against 52,657 in Q2 (up 4.2 percent)
  3. 03The share answered within 15 seconds is computed from the counts: 86.6 percent in Q3 against 90.5 percent in Q2, below the 90 percent standard in Q3 (monthly: Apr 91.2, May 90.6, Jun 89.8, Jul 86.4, Aug 84.9, Sep 88.7)
  4. 04The abandoned-call rate is correct: 3.7 percent in Q3 against 3.3 percent in Q2
  5. 05Staffing and overtime are correct: 7 of 48 positions vacant at the end of September; Q3 overtime 8,750 hours, about $360,937.50 at $41.25 an hour (Q2 7,855 hours)
  6. 06The report does not repeat the supervisors' claim that answer times improved: it reports that they declined (or flags that the note conflicts with the data), and ties the August dip to the outage and the CAD cutover only as the notes support
  7. 07Details the request doesn't give (the director's and Board chair's names, signatures, addresses) are clearly labeled blanks, not invented
  8. 08A one-page transmittal letter from the director to the Board chair is delivered as a separate Word document
  9. 09The report is professional, readable and ready for the Board once labeled blanks are filled

Agencies that use it

  • Agriculture & Rural Services
  • Housing & Economic Development
  • Courts & Justice Administration
  • Finance, Insurance & Retirement
  • Transportation, Facilities & Infrastructure
  • Health, Human Services & Veterans
  • Licensing & Professional Regulation
  • Public Safety, Corrections & Emergency Management
  • Energy & Utilities
  • Environment, Water & Natural Resources
  • Education & Workforce

Example tasks

What the systems were asked to do

Real tasks from this benchmark. Every system got the same prompt and data.

Reports and minutes

Reports, briefings and correspondence

I'm the deputy director of our county's Emergency Communications (911) Center. Write our third-quarter 2026 performance report for the Board of County…

Full task
I'm the deputy director of our county's Emergency Communications (911) Center. Write our third-quarter 2026 performance report for the Board of County Commissioners' December 3, 2026 meeting, as a Word document: an executive summary, the quarter's results compared with the second quarter, a table and at least one chart, what explains the changes, and recommendations. Our standard is to answer 90 percent of 911 calls within 15 seconds. Also write a one-page transmittal letter from our director to the Board chair, as a separate Word document.
Use the data and notes below. Keep every figure accurate to the record.

month,calls_received,answered_within_15_seconds,abandoned_calls,telecommunicator_positions_filled,overtime_hours,text_to_911_sessions
2026-04,16902,15414,541,40,2410,168
2026-05,17640,15983,582,40,2655,181
2026-06,18115,16267,616,39,2790,197
2026-07,18412,15908,662,39,2960,212
2026-08,19060,16182,781,38,3310,236
2026-09,17385,15420,574,41,2480,251

Shift supervisor notes for the report (from the operations meeting, October 6, 2026):
- "Answer times improved this quarter thanks to the new CAD system." The new computer-aided dispatch (CAD) system went live August 4, 2026.
- From August 19 to 21, a carrier outage sent our calls to the backup center at the sheriff's office for about 41 hours.
- Two trainees finished certification and moved to the floor on September 8.
- We have 48 authorized telecommunicator positions.
- Overtime is paid at an average of $41.25 an hour.
- The commission has asked whether we should keep paying overtime or fund the 4 trainee positions in next year's budget.

Task types

What this benchmark covers

Reports and minutes2 tasks
Reports and minutes that match the underlying records.
Briefings and talking points2 tasks
Briefings for leadership with every figure sourced.
Web content and letters1 task
Public pages and letters in plain language, accurate to policy.

Method

How we test

  1. 01

    Real agency deliverables

    One task per deliverable use case: trackers, dashboards, permit systems, forms, decks, workbooks, letters and one-pagers. Each carries its own rules and data, with problems planted where a careful reviewer would catch them.

  2. 02

    Every system can make files

    ChatGPT Astra 6.2, Claude Fable 5.2, Grok 4.7 and Gemini 4 Carbon run on their makers' own services with web search, high reasoning effort and a code tool whose files are what the user receives. When their code crashed or sent no files, they got up to two follow-ups. Justinian ran as shipped, with none.

  3. 03

    Graded as delivered

    Every file is rendered as the requester would open it and every app is opened in a browser. Code or instructions the user would still have to run, deploy or schedule do not count. Bill numbers and rule sections are checked against the official record, and every regulatory call is checked against the statute.

  4. 04

    Three blind judges

    Judges from OpenAI, Anthropic and Google, none of them a competing model, score each deliverable as the user would see it and rank all five systems with names masked. Every system is graded with the same checklist and scoring guide.

Briefing

Test Justinian on your agency's own work.

Request a briefing →

This page’s interactive features did not load. Reloading usually fixes it. You can email sales@vulcan.ai for demos or support@vulcan.ai for help.