All benchmarks
Permits and licensing

A working permit system built from the governing rule: application form, eligibility and completeness checks, fee calculation, reviewer queue and decision notices.

Deliverable
A working permit system, run against test applications

Permitting system, end to end · overall

Deliverable quality score

  • Justinian3.2C+71.7%
  • ChatGPTAstra 6.2C-60.3%
  • ClaudeFable 5.2D+53.3%
  • Grok4.7D+55.3%
  • Gemini4 CarbonC-57%

Checklist

What the judges checked

Written before any system ran.

  1. 01A working application form is delivered that a person can open and submit
  2. 02Completeness checks enforce the filing deadlines (30 days, 60 days over 1,000) and required items, including the higher insurance minimum with alcohol
  3. 03Fees calculate correctly: e.g., a nonprofit, 600 attendees, 2 blocks closed, alcohol: (150 + 800 + 600 + 250) x 50% = $900
  4. 04Routing sends applications to Public Works (street closure), Police (alcohol) and Fire Marshal (over 500) as the rules require
  5. 05Decision notices include the reason for denial and the 15-day appeal right to the City Manager
  6. 06Applications and decisions persist so a reviewer queue actually works across sessions, not just a static mockup

Example tasks

What the systems were asked to do

Real tasks from this benchmark. Every system got the same prompt and data.

Building permits

Permitting system, end to end

Build a working special event permit system for our city from the rules below: an online application form, checks for eligibility and completeness, automatic…

Full task
Build a working special event permit system for our city from the rules below: an online application form, checks for eligibility and completeness, automatic fee calculation, a reviewer queue that routes each application to the departments that must sign off, and decision notices (approval or denial with reason and appeal rights). Share it so anyone with the link can view it, and give me the link.

City of Riverton special event permit rules (Riverton Code ch. 9.40):
- Applies to any event on public property or a public street with 75 or more expected attendees.
- Application must be filed at least 30 days before the event; events over 1,000 attendees need 60 days.
- Required: applicant name and contact, event name, date and hours, location, expected attendance, site map upload, certificate of liability insurance of at least $1,000,000 (at least $2,000,000 if alcohol is served).
- Street closures require a traffic control plan and Public Works sign-off.
- Alcohol requires Police sign-off and proof of a state temporary alcohol license.
- Over 500 attendees requires Fire Marshal sign-off.
- Fees: base $150; plus $2 per expected attendee over 200; street closure $300 per block; alcohol $250. Nonprofit organizations pay 50% of the total.
- The city must issue a decision within 10 business days of a complete application. Denials must state the reason and the right to appeal to the City Manager within 15 days.

Task types

What this benchmark covers

Building permits
Residential and commercial permits with plan review, zoning checks and fee schedules.
Environmental permits
Air, water and waste permits with completeness rules and public notice.
Business and professional licenses
New and renewal licenses with eligibility, reciprocity and continuing education checks.
Special event and right-of-way permits
Time-limited permits with insurance, traffic control and sign-off from several departments.

Method

How we test

  1. 01

    Real agency deliverables

    One task per deliverable use case: trackers, dashboards, permit systems, forms, decks, workbooks, letters and one-pagers. Each carries its own rules and data, with problems planted where a careful reviewer would catch them.

  2. 02

    Every system can make files

    ChatGPT Astra 6.2, Claude Fable 5.2, Grok 4.7 and Gemini 4 Carbon run on their makers' own services with web search, high reasoning effort and a code tool whose files are what the user receives. When their code crashed or sent no files, they got up to two follow-ups. Justinian ran as shipped, with none.

  3. 03

    Graded as delivered

    Every file is rendered as the requester would open it and every app is opened in a browser. Code or instructions the user would still have to run, deploy or schedule do not count. Bill numbers and rule sections are checked against the official record, and every regulatory call is checked against the statute.

  4. 04

    Three blind judges

    Judges from OpenAI, Anthropic and Google, none of them a competing model, score each deliverable as the user would see it and rank all five systems with names masked. Every system is graded with the same checklist and scoring guide.

Briefing

Test Justinian on your agency's own work.

Request a briefing →

This page’s interactive features did not load. Reloading usually fixes it. You can email sales@vulcan.ai for demos or support@vulcan.ai for help.