All benchmarks
Workforce and operations

A designed one-page brief for a director or legislator, with the key figures charted.

Deliverable
A designed one-page PDF

Leadership one-pager · overall

Deliverable quality score

  • Justinian3.2A93.3%
  • ChatGPTAstra 6.2B+85%
  • ClaudeFable 5.2B+85.7%
  • Grok4.7D-45%
  • Gemini4 CarbonD-44.3%

Checklist

What the judges checked

Written before any system ran.

  1. 01A PDF of exactly one page is delivered
  2. 02At least one real chart of the data is included and it is accurate
  3. 03All figures used match the data provided
  4. 04A clear recommendation (statewide online ordering) with cost and the unspent-funds argument is stated
  5. 05Design is clean and professional; text is readable at print size

Example tasks

What the systems were asked to do

Real tasks from this benchmark. Every system got the same prompt and data.

Program briefs

Leadership one-pager

Make a designed one-page PDF brief for our Secretary of Health, who meets the Governor tomorrow about the WIC program. Data: participants 2023 112,400; 2024…

Full task
Make a designed one-page PDF brief for our Secretary of Health, who meets the Governor tomorrow about the WIC program. Data: participants 2023 112,400; 2024 108,900; 2025 104,300; 2026 101,700. Eligible but not enrolled 2026: 61,000 (est.). Redemption rate of food benefits 2026: 71% (national 79%). Clinics: 96, down from 104 in 2023. Pilot of online ordering in 6 counties raised redemption from 68% to 80% in 9 months. Cost to expand online ordering statewide: $1.9 million one-time, $420,000 per year. Federal food funds unspent in FY2026 because of low redemption: $14.2 million.
Include the key figures charted and a clear recommendation.

Task types

What this benchmark covers

Program briefs
What a program does, what it costs and what it achieved.
Issue briefs
A decision a leader faces, the options and a recommendation.

Method

How we test

  1. 01

    Real agency deliverables

    One task per deliverable use case: trackers, dashboards, permit systems, forms, decks, workbooks, letters and one-pagers. Each carries its own rules and data, with problems planted where a careful reviewer would catch them.

  2. 02

    Every system can make files

    ChatGPT Astra 6.2, Claude Fable 5.2, Grok 4.7 and Gemini 4 Carbon run on their makers' own services with web search, high reasoning effort and a code tool whose files are what the user receives. When their code crashed or sent no files, they got up to two follow-ups. Justinian ran as shipped, with none.

  3. 03

    Graded as delivered

    Every file is rendered as the requester would open it and every app is opened in a browser. Code or instructions the user would still have to run, deploy or schedule do not count. Bill numbers and rule sections are checked against the official record, and every regulatory call is checked against the statute.

  4. 04

    Three blind judges

    Judges from OpenAI, Anthropic and Google, none of them a competing model, score each deliverable as the user would see it and rank all five systems with names masked. Every system is graded with the same checklist and scoring guide.

Briefing

Test Justinian on your agency's own work.

Request a briefing →

This page’s interactive features did not load. Reloading usually fixes it. You can email sales@vulcan.ai for demos or support@vulcan.ai for help.