Reports, minutes, briefings, talking points, web content and letters, accurate to the record.
- Deliverable
- A finished document ready for review
Reports, briefings and correspondence · overall
Deliverable quality score
- Justinian3.2A92%
- ChatGPTAstra 6.2B78%
- ClaudeFable 5.2A-87.3%
- Grok4.7C+70%
- Gemini4 CarbonC+67.7%
Checklist
What the judges checked
Written before any system ran.
- 01A Word report is delivered with an executive summary, a comparison with the second quarter, at least one table and one chart, explanations and recommendations
- 02Call volume is correct: 54,857 calls in Q3 against 52,657 in Q2 (up 4.2 percent)
- 03The share answered within 15 seconds is computed from the counts: 86.6 percent in Q3 against 90.5 percent in Q2, below the 90 percent standard in Q3 (monthly: Apr 91.2, May 90.6, Jun 89.8, Jul 86.4, Aug 84.9, Sep 88.7)
- 04The abandoned-call rate is correct: 3.7 percent in Q3 against 3.3 percent in Q2
- 05Staffing and overtime are correct: 7 of 48 positions vacant at the end of September; Q3 overtime 8,750 hours, about $360,937.50 at $41.25 an hour (Q2 7,855 hours)
- 06The report does not repeat the supervisors' claim that answer times improved: it reports that they declined (or flags that the note conflicts with the data), and ties the August dip to the outage and the CAD cutover only as the notes support
- 07Details the request doesn't give (the director's and Board chair's names, signatures, addresses) are clearly labeled blanks, not invented
- 08A one-page transmittal letter from the director to the Board chair is delivered as a separate Word document
- 09The report is professional, readable and ready for the Board once labeled blanks are filled
Agencies that use it
- Agriculture & Rural Services
- Housing & Economic Development
- Courts & Justice Administration
- Finance, Insurance & Retirement
- Transportation, Facilities & Infrastructure
- Health, Human Services & Veterans
- Licensing & Professional Regulation
- Public Safety, Corrections & Emergency Management
- Energy & Utilities
- Environment, Water & Natural Resources
- Education & Workforce
Example tasks
What the systems were asked to do
Real tasks from this benchmark. Every system got the same prompt and data.
Reports and minutes
Reports, briefings and correspondence
I'm the deputy director of our county's Emergency Communications (911) Center. Write our third-quarter 2026 performance report for the Board of County…
Full task
I'm the deputy director of our county's Emergency Communications (911) Center. Write our third-quarter 2026 performance report for the Board of County Commissioners' December 3, 2026 meeting, as a Word document: an executive summary, the quarter's results compared with the second quarter, a table and at least one chart, what explains the changes, and recommendations. Our standard is to answer 90 percent of 911 calls within 15 seconds. Also write a one-page transmittal letter from our director to the Board chair, as a separate Word document. Use the data and notes below. Keep every figure accurate to the record. month,calls_received,answered_within_15_seconds,abandoned_calls,telecommunicator_positions_filled,overtime_hours,text_to_911_sessions 2026-04,16902,15414,541,40,2410,168 2026-05,17640,15983,582,40,2655,181 2026-06,18115,16267,616,39,2790,197 2026-07,18412,15908,662,39,2960,212 2026-08,19060,16182,781,38,3310,236 2026-09,17385,15420,574,41,2480,251 Shift supervisor notes for the report (from the operations meeting, October 6, 2026): - "Answer times improved this quarter thanks to the new CAD system." The new computer-aided dispatch (CAD) system went live August 4, 2026. - From August 19 to 21, a carrier outage sent our calls to the backup center at the sheriff's office for about 41 hours. - Two trainees finished certification and moved to the floor on September 8. - We have 48 authorized telecommunicator positions. - Overtime is paid at an average of $41.25 an hour. - The commission has asked whether we should keep paying overtime or fund the 4 trainee positions in next year's budget.
Task types
What this benchmark covers
- Reports and minutes2 tasks
- Reports and minutes that match the underlying records.
- Briefings and talking points2 tasks
- Briefings for leadership with every figure sourced.
- Web content and letters1 task
- Public pages and letters in plain language, accurate to policy.
Method
How we test
01
Real agency deliverables
One task per deliverable use case: trackers, dashboards, permit systems, forms, decks, workbooks, letters and one-pagers. Each carries its own rules and data, with problems planted where a careful reviewer would catch them.
02
Every system can make files
ChatGPT Astra 6.2, Claude Fable 5.2, Grok 4.7 and Gemini 4 Carbon run on their makers' own services with web search, high reasoning effort and a code tool whose files are what the user receives. When their code crashed or sent no files, they got up to two follow-ups. Justinian ran as shipped, with none.
03
Graded as delivered
Every file is rendered as the requester would open it and every app is opened in a browser. Code or instructions the user would still have to run, deploy or schedule do not count. Bill numbers and rule sections are checked against the official record, and every regulatory call is checked against the statute.
04
Three blind judges
Judges from OpenAI, Anthropic and Google, none of them a competing model, score each deliverable as the user would see it and rank all five systems with names masked. Every system is graded with the same checklist and scoring guide.
More in Workforce and operations
All benchmarksBriefing
