An inventory of a chapter's provisions that flags outdated, duplicative and unauthorized ones, with a tracked-change redline and a summary memo.
- Deliverable
- An inventory workbook, a tracked-change redline and a memo
Regulation streamlining · overall
Deliverable quality score
- Justinian3.2B78%
- ChatGPTAstra 6.2C+68.3%
- ClaudeFable 5.2C-58.3%
- Gemini4 CarbonF35.3%
Checklist
What the judges checked
Written before any system ran.
- 01An inventory spreadsheet is delivered and opens, with one row per section and all requested columns
- 02The inventory covers Chapter 60 broadly (dozens of sections, not a handful) with real section numbers in 16 TAC Chapter 60
- 03Each flag has a specific reason tied to the section's text or authority, not boilerplate
- 04A Word document is delivered whose amendments are real tracked changes (insertions/deletions), not colored or struck text
- 05The tracked changes are made to the actual text of real Chapter 60 sections
- 06A memo is delivered, about two pages, with findings ordered by priority
Example tasks
What the systems were asked to do
Real tasks from this benchmark. Every system got the same prompt and data.
Provision inventory
Regulation streamlining
Our agency is doing a regulatory review of Texas Administrative Code, Title 16, Part 4, Chapter 60 (the Texas Department of Licensing and Regulation's…
Full task
Our agency is doing a regulatory review of Texas Administrative Code, Title 16, Part 4, Chapter 60 (the Texas Department of Licensing and Regulation's procedural rules). Deliver: 1. An inventory spreadsheet with one row per section in Chapter 60: section number, title, statutory authority cited, date last amended if available, and a flag (Keep / Update / Consolidate / Repeal) with the reason. 2. A Word document with your proposed amendments to the five highest-priority sections as real tracked changes (insertions and deletions I can accept or reject in Word). 3. A two-page memo to the executive director summarizing the findings by priority.
Task types
What this benchmark covers
- Provision inventory
- Every section in the chapter with its authority, last amendment and a flag with the reason.
- Tracked-change redline
- Proposed amendments as real tracked changes in Word.
- Summary memo
- Findings by priority for the agency head.
Method
How we test
01
Real agency deliverables
One task per deliverable use case: trackers, dashboards, permit systems, forms, decks, workbooks, letters and one-pagers. Each carries its own rules and data, with problems planted where a careful reviewer would catch them.
02
Every system can make files
ChatGPT Astra 6.2, Claude Fable 5.2, Grok 4.7 and Gemini 4 Carbon run on their makers' own services with web search, high reasoning effort and a code tool whose files are what the user receives. When their code crashed or sent no files, they got up to two follow-ups. Justinian ran as shipped, with none.
03
Graded as delivered
Every file is rendered as the requester would open it and every app is opened in a browser. Code or instructions the user would still have to run, deploy or schedule do not count. Bill numbers and rule sections are checked against the official record, and every regulatory call is checked against the statute.
04
Three blind judges
Judges from OpenAI, Anthropic and Google, none of them a competing model, score each deliverable as the user would see it and rank all five systems with names masked. Every system is graded with the same checklist and scoring guide.
More in Research and rules
All benchmarksBriefing
