All benchmarks
Grants, contracts and budgets

Spending against appropriation by program and month from the agency's ledger, with variances flagged.

Deliverable
A shareable dashboard

Budget dashboard · overall

Deliverable quality score

  • Justinian3.2
    97%
  • ChatGPTAstra 6.2
    73.3%
  • ClaudeFable 5.2
    75.3%
  • Grok4.7
    74%
  • Gemini4 Carbon
    69.3%

Checklist

What the judges checked

Written before any system ran.

  1. 01A dashboard is delivered that a person can open
  2. 02Year-to-date spending by program is computed correctly from the ledger
  3. 03Exactly the programs more than 10% off pace are flagged (Public health, Road maintenance, Senior services)
  4. 04Monthly trend by program is shown
  5. 05Program and fund filters work
  6. 06It is clear and presentable to elected officials

Example tasks

What the systems were asked to do

Real tasks from this benchmark. Every system got the same prompt and data.

Spend against budget

Budget dashboard

Build a budget dashboard for our county commissioners from the ledger below (January to September 2026). Show spending against budget by program and by month,…

Full task
Build a budget dashboard for our county commissioners from the ledger below (January to September 2026). Show spending against budget by program and by month, year to date, and flag any program whose year-to-date spending is more than 10% above or below its pro-rated budget (9/12 of the annual budget). Include filters for program and fund. Share it so anyone with the link can view it, and give me the link.

program,fund,month,approved_budget_annual,spent
Road maintenance,General,2026-01,900000,51904
Road maintenance,General,2026-02,900000,49257
Road maintenance,General,2026-03,900000,47592
Road maintenance,General,2026-04,900000,55369
Road maintenance,General,2026-05,900000,45583
Road maintenance,General,2026-06,900000,53941
Road maintenance,General,2026-07,900000,57735
Road maintenance,General,2026-08,900000,55247
Road maintenance,General,2026-09,900000,50504
Parks,Special Revenue,2026-01,900000,71035
Parks,Special Revenue,2026-02,900000,68932
Parks,Special Revenue,2026-03,900000,85642
Parks,Special Revenue,2026-04,900000,83768
Parks,Special Revenue,2026-05,900000,85024
Parks,Special Revenue,2026-06,900000,76681
Parks,Special Revenue,2026-07,900000,88386
Parks,Special Revenue,2026-08,900000,83459
Parks,Special Revenue,2026-09,900000,86573
Libraries,General,2026-01,900000,72748
Libraries,General,2026-02,900000,77004
Libraries,General,2026-03,900000,80462
Libraries,General,2026-04,900000,87097
Libraries,General,2026-05,900000,86354
Libraries,General,2026-06,900000,76949
Libraries,General,2026-07,900000,88468
Libraries,General,2026-08,900000,72316
Libraries,General,2026-09,900000,86848
Public health,General,2026-01,900000,49034
Public health,General,2026-02,900000,56813
Public health,General,2026-03,900000,47036
Public health,General,2026-04,900000,58187
Public health,General,2026-05,900000,51057
Public health,General,2026-06,900000,57372
Public health,General,2026-07,900000,54382
Public health,General,2026-08,900000,47399
Public health,General,2026-09,900000,55484
Code enforcement,General,2026-01,900000,69907
Code enforcement,General,2026-02,900000,68419
Code enforcement,General,2026-03,900000,68820
Code enforcement,General,2026-04,900000,84993
Code enforcement,General,2026-05,900000,79246
Code enforcement,General,2026-06,900000,64239
Code enforcement,General,2026-07,900000,69847
Code enforcement,General,2026-08,900000,77654
Code enforcement,General,2026-09,900000,85962
Senior services,Special Revenue,2026-01,3100000,354544
Senior services,Special Revenue,2026-02,3100000,358931
Senior services,Special Revenue,2026-03,3100000,315481
Senior services,Special Revenue,2026-04,3100000,354262
Senior services,Special Revenue,2026-05,3100000,320732
Senior services,Special Revenue,2026-06,3100000,336615
Senior services,Special Revenue,2026-07,3100000,294630
Senior services,Special Revenue,2026-08,3100000,280064
Senior services,Special Revenue,2026-09,3100000,298766

Task types

What this benchmark covers

Spend against budget
Year to date by program and fund.
Variance flags
Lines over or under plan beyond a set threshold.

Method

How we test

  1. 01

    Real agency deliverables

    One task per deliverable use case: trackers, dashboards, permit systems, forms, decks, workbooks, letters and one-pagers. Each carries its own rules and data, with problems planted where a careful reviewer would catch them.

  2. 02

    Every system can make files

    ChatGPT Astra 6.2, Claude Fable 5.2, Grok 4.7 and Gemini 4 Carbon run on their makers' own services with web search, high reasoning effort and a code tool whose files are what the user receives. When their code crashed or sent no files, they got up to two follow-ups. Justinian ran as shipped, with none.

  3. 03

    Graded as delivered

    Every file is rendered as the requester would open it and every app is opened in a browser. Code or instructions the user would still have to run, deploy or schedule do not count. Bill numbers and rule sections are checked against the official record, and every regulatory call is checked against the statute.

  4. 04

    Three blind judges

    Judges from OpenAI, Anthropic and Google, none of them a competing model, score each deliverable as the user would see it and rank all five systems with names masked. Every system is graded with the same checklist and scoring guide.

Briefing

Test Justinian on your agency's own work.

Request a briefing →

This page’s interactive features did not load. Reloading usually fixes it. You can email sales@vulcan.ai for demos or support@vulcan.ai for help.