All benchmarks
Research and rules

An inventory of a chapter's provisions that flags outdated, duplicative and unauthorized ones, with a tracked-change redline and a summary memo.

Deliverable
An inventory workbook, a tracked-change redline and a memo

Regulation streamlining · overall

Deliverable quality score

  • Justinian3.2B78%
  • ChatGPTAstra 6.2C+68.3%
  • ClaudeFable 5.2C-58.3%
  • Gemini4 CarbonF35.3%

Checklist

What the judges checked

Written before any system ran.

  1. 01An inventory spreadsheet is delivered and opens, with one row per section and all requested columns
  2. 02The inventory covers Chapter 60 broadly (dozens of sections, not a handful) with real section numbers in 16 TAC Chapter 60
  3. 03Each flag has a specific reason tied to the section's text or authority, not boilerplate
  4. 04A Word document is delivered whose amendments are real tracked changes (insertions/deletions), not colored or struck text
  5. 05The tracked changes are made to the actual text of real Chapter 60 sections
  6. 06A memo is delivered, about two pages, with findings ordered by priority

Example tasks

What the systems were asked to do

Real tasks from this benchmark. Every system got the same prompt and data.

Provision inventory

Regulation streamlining

Our agency is doing a regulatory review of Texas Administrative Code, Title 16, Part 4, Chapter 60 (the Texas Department of Licensing and Regulation's…

Full task
Our agency is doing a regulatory review of Texas Administrative Code, Title 16, Part 4, Chapter 60 (the Texas Department of Licensing and Regulation's procedural rules).
Deliver:
1. An inventory spreadsheet with one row per section in Chapter 60: section number, title, statutory authority cited, date last amended if available, and a flag (Keep / Update / Consolidate / Repeal) with the reason.
2. A Word document with your proposed amendments to the five highest-priority sections as real tracked changes (insertions and deletions I can accept or reject in Word).
3. A two-page memo to the executive director summarizing the findings by priority.

Task types

What this benchmark covers

Provision inventory
Every section in the chapter with its authority, last amendment and a flag with the reason.
Tracked-change redline
Proposed amendments as real tracked changes in Word.
Summary memo
Findings by priority for the agency head.

Method

How we test

  1. 01

    Real agency deliverables

    One task per deliverable use case: trackers, dashboards, permit systems, forms, decks, workbooks, letters and one-pagers. Each carries its own rules and data, with problems planted where a careful reviewer would catch them.

  2. 02

    Every system can make files

    ChatGPT Astra 6.2, Claude Fable 5.2, Grok 4.7 and Gemini 4 Carbon run on their makers' own services with web search, high reasoning effort and a code tool whose files are what the user receives. When their code crashed or sent no files, they got up to two follow-ups. Justinian ran as shipped, with none.

  3. 03

    Graded as delivered

    Every file is rendered as the requester would open it and every app is opened in a browser. Code or instructions the user would still have to run, deploy or schedule do not count. Bill numbers and rule sections are checked against the official record, and every regulatory call is checked against the statute.

  4. 04

    Three blind judges

    Judges from OpenAI, Anthropic and Google, none of them a competing model, score each deliverable as the user would see it and rank all five systems with names masked. Every system is graded with the same checklist and scoring guide.

Briefing

Test Justinian on your agency's own work.

Request a briefing →

This page’s interactive features did not load. Reloading usually fixes it. You can email sales@vulcan.ai for demos or support@vulcan.ai for help.