All benchmarks
Compliance and audit

Schedules inspections from a facility list, records findings and sends follow-up notices when corrections are overdue.

Deliverable
A working tracker with scheduled follow-ups

Inspection tracker · overall

Deliverable quality score

  • Justinian3.2
    92.3%
  • ChatGPTAstra 6.2
    60.3%
  • ClaudeFable 5.2
    60.3%
  • Grok4.7
    56.3%
  • Gemini4 Carbon
    39.7%

Checklist

What the judges checked

Written before any system ran.

  1. 01A tracker is delivered that a person can open and use
  2. 02Next due dates are computed from the frequency rules, and exactly the overdue facilities are flagged (15 overdue: F-1001, F-1003, F-1004, F-1007, F-1009, F-1010, F-1012, F-1013, F-1015, F-1017, F-1018, F-1019, F-1020, F-1023, F-1025)
  3. 03Inspectors can record findings with violation, code cited and correction deadline, and the records persist
  4. 04Automatic follow-up notices for missed correction deadlines are actually set up to run, not only described
  5. 05The interface is clear enough for an inspector to use without training

Example tasks

What the systems were asked to do

Real tasks from this benchmark. Every system got the same prompt and data.

Scheduling

Inspection tracker

Build an inspection tracker for our county health department from the facility list below. Inspection frequency by type: restaurants every 12 months, food…

Full task
Build an inspection tracker for our county health department from the facility list below. Inspection frequency by type: restaurants every 12 months, food trucks every 6 months, child care centers every 6 months, swimming pools every 12 months, hotels every 24 months. Today is November 18, 2026.
It should show each facility's next due date, flag overdue ones, let inspectors record findings (violation, code cited, correction deadline), and send a follow-up notice automatically when a correction deadline passes without the violation being closed. Share it so anyone with the link can view it, and give me the link.

facility_id,name,type,last_inspection
F-1001,Oak Street 1,Food truck,2025-11-12
F-1002,Main Kids 2,Restaurant,2026-03-13
F-1003,Main Aquatics 3,Swimming pool,2024-11-08
F-1004,Cedar Cafe 4,Food truck,2026-03-21
F-1005,Elm Street 5,Restaurant,2026-03-09
F-1006,River Kids 6,Child care center,2026-10-23
F-1007,Oak Market 7,Child care center,2024-09-28
F-1008,Main Grill 8,Hotel,2026-03-30
F-1009,River Street 9,Food truck,2025-03-11
F-1010,River Street 10,Swimming pool,2025-08-18
F-1011,Bay Market 11,Swimming pool,2026-04-28
F-1012,River Cafe 12,Child care center,2025-01-31
F-1013,Bay Aquatics 13,Swimming pool,2025-10-10
F-1014,Oak Aquatics 14,Child care center,2026-09-01
F-1015,Elm Aquatics 15,Child care center,2025-11-13
F-1016,Cedar Aquatics 16,Hotel,2025-10-09
F-1017,Cedar Kids 17,Food truck,2025-09-17
F-1018,Bay Street 18,Child care center,2025-01-26
F-1019,Pine Cafe 19,Food truck,2025-03-31
F-1020,Cedar Grill 20,Swimming pool,2025-07-04
F-1021,Oak Aquatics 21,Hotel,2026-05-28
F-1022,Main Inn 22,Hotel,2026-02-24
F-1023,Oak Kids 23,Swimming pool,2025-04-30
F-1024,Elm Grill 24,Swimming pool,2026-07-26
F-1025,Oak Street 25,Restaurant,2025-11-10

Task types

What this benchmark covers

Scheduling
Inspection frequency set by the rule for each facility type.
Findings and corrections
Violations recorded with the provision cited and a correction deadline.
Follow-up notices
Notices sent automatically when a correction is overdue.

Method

How we test

  1. 01

    Real agency deliverables

    One task per deliverable use case: trackers, dashboards, permit systems, forms, decks, workbooks, letters and one-pagers. Each carries its own rules and data, with problems planted where a careful reviewer would catch them.

  2. 02

    Every system can make files

    ChatGPT Astra 6.2, Claude Fable 5.2, Grok 4.7 and Gemini 4 Carbon run on their makers' own services with web search, high reasoning effort and a code tool whose files are what the user receives. When their code crashed or sent no files, they got up to two follow-ups. Justinian ran as shipped, with none.

  3. 03

    Graded as delivered

    Every file is rendered as the requester would open it and every app is opened in a browser. Code or instructions the user would still have to run, deploy or schedule do not count. Bill numbers and rule sections are checked against the official record, and every regulatory call is checked against the statute.

  4. 04

    Three blind judges

    Judges from OpenAI, Anthropic and Google, none of them a competing model, score each deliverable as the user would see it and rank all five systems with names masked. Every system is graded with the same checklist and scoring guide.

More in Compliance and audit

All benchmarks

Briefing

Test Justinian on your agency's own work.

Request a briefing →

This page’s interactive features did not load. Reloading usually fixes it. You can email sales@vulcan.ai for demos or support@vulcan.ai for help.