All benchmarks
Research and rules

A live tracker of the bills on a topic, with status, last action and next hearing, and an alert when any of them moves.

Deliverable
A working tracker and a scheduled alert

Bill tracker · overall

Deliverable quality score

  • Justinian3.2
    90%
  • ChatGPTAstra 6.2
    70.7%
  • ClaudeFable 5.2
    51.7%
  • Grok4.7
    11.7%No deliverable
  • Gemini4 Carbon
    63.3%

Checklist

What the judges checked

Written before any system ran.

  1. 01A spreadsheet file is delivered and opens, with one row per bill and all seven requested fields
  2. 02At least 15 distinct Pennsylvania 2025-2026 bills on AI are listed
  3. 03Bill numbers, sponsors and statuses shown are consistent with the official record (judge flags any that look invented or wrong)
  4. 04Last-action dates are recent (2026) where the bill moved this year, not stale 2025 placeholders
  5. 05A dashboard is delivered that a person can open and filter by status and by topic
  6. 06The weekly alert is actually set up to run (scheduled), not only described or left as code for the user to deploy

Example tasks

What the systems were asked to do

Real tasks from this benchmark. Every system got the same prompt and data.

Tracker build

Bill tracker

I track artificial intelligence legislation for a Pennsylvania state agency. Build me a tracker of every bill in the Pennsylvania General Assembly's 2025-2026…

Full task
I track artificial intelligence legislation for a Pennsylvania state agency. Build me a tracker of every bill in the Pennsylvania General Assembly's 2025-2026 session that would regulate artificial intelligence (for example deepfakes, AI in insurance or health care decisions, AI chatbots, government use of AI, AI-generated election content).
For each bill give: bill number, short title, prime sponsor, topic, current status, date of last action, and a link to the official bill page.
Deliver (1) a spreadsheet I can keep, (2) a dashboard I can filter by status and topic (Share it so anyone with the link can view it, and give me the link.), and (3) a weekly alert that tells me which of these bills had a new action that week. Set the alert up; don't just describe it.

Task types

What this benchmark covers

Tracker build
Every bill on the topic in the current session, with sponsor, status, last action and next hearing.
Change alerts
A scheduled check that sends notice when a tracked bill is amended, scheduled or voted on.
Weekly summary
A one-page summary of what moved, generated from the tracker.

Method

How we test

  1. 01

    Real agency deliverables

    One task per deliverable use case: trackers, dashboards, permit systems, forms, decks, workbooks, letters and one-pagers. Each carries its own rules and data, with problems planted where a careful reviewer would catch them.

  2. 02

    Every system can make files

    ChatGPT Astra 6.2, Claude Fable 5.2, Grok 4.7 and Gemini 4 Carbon run on their makers' own services with web search, high reasoning effort and a code tool whose files are what the user receives. When their code crashed or sent no files, they got up to two follow-ups. Justinian ran as shipped, with none.

  3. 03

    Graded as delivered

    Every file is rendered as the requester would open it and every app is opened in a browser. Code or instructions the user would still have to run, deploy or schedule do not count. Bill numbers and rule sections are checked against the official record, and every regulatory call is checked against the statute.

  4. 04

    Three blind judges

    Judges from OpenAI, Anthropic and Google, none of them a competing model, score each deliverable as the user would see it and rank all five systems with names masked. Every system is graded with the same checklist and scoring guide.

Briefing

Test Justinian on your agency's own work.

Request a briefing →

This page’s interactive features did not load. Reloading usually fixes it. You can email sales@vulcan.ai for demos or support@vulcan.ai for help.