All benchmarks
Permits and licensing

Completeness reviews, draft permits, deficiency letters, conditions and comment responses, checked against the program's rules.

Deliverable
A draft permit document ready for reviewer edits

Permit review and drafting · overall

Deliverable quality score

  • Justinian3.2A92.7%
  • ChatGPTAstra 6.2C64.3%
  • ClaudeFable 5.2A93.3%
  • Grok4.7D51.7%
  • Gemini4 CarbonC-59%

Checklist

What the judges checked

Written before any system ran.

  1. 01A Word document is delivered and opens, formatted as a city letter
  2. 02The decision is a denial (or not-complete notice) because the application was filed fewer than 60 days before an event over 1,000 attendees
  3. 03It also identifies the insufficient insurance ($1M provided, $2M required with alcohol) and the missing alcohol license
  4. 04It states the fee correctly if refiled: (150 + 2,000 + 900 + 250) x 50% = $1,650
  5. 05It states the right to appeal to the City Manager within 15 days
  6. 06Tone and format are ready to send without edits

Agencies that use it

  • Environment, Water & Natural Resources
  • Energy & Utilities
  • Transportation, Facilities & Infrastructure

In the field

What agencies did with it

Results agencies reported from their own work in Justinian.

Louisiana Department of Environmental Quality

Personnel

Reviewing permits faster

per permit form rules check
32 min
per permit form rules check
pages of comment responses in 43 minutes
1,886
pages of comment responses in 43 minutes

Example tasks

What the systems were asked to do

Real tasks from this benchmark. Every system got the same prompt and data.

Completeness review

Permit review and drafting

Draft the decision letter (Word document) for this application under the rules below. Application: Riverside Arts Collective (a registered nonprofit), "Night…

Full task
Draft the decision letter (Word document) for this application under the rules below. Application: Riverside Arts Collective (a registered nonprofit), "Night Market on Main," Saturday, December 12, 2026, 4 to 11 p.m., Main Street between 2nd and 5th (3 blocks closed), expected attendance 1,200, beer garden planned. Filed November 18, 2026. Insurance certificate attached: $1,000,000. Traffic control plan attached. State temporary alcohol license: applied for, not yet issued.
Decide it correctly under the rules, explain each reason with the code section, state the fee that would apply, and include the appeal rights.

City of Riverton special event permit rules (Riverton Code ch. 9.40):
- Applies to any event on public property or a public street with 75 or more expected attendees.
- Application must be filed at least 30 days before the event; events over 1,000 attendees need 60 days.
- Required: applicant name and contact, event name, date and hours, location, expected attendance, site map upload, certificate of liability insurance of at least $1,000,000 (at least $2,000,000 if alcohol is served).
- Street closures require a traffic control plan and Public Works sign-off.
- Alcohol requires Police sign-off and proof of a state temporary alcohol license.
- Over 500 attendees requires Fire Marshal sign-off.
- Fees: base $150; plus $2 per expected attendee over 200; street closure $300 per block; alcohol $250. Nonprofit organizations pay 50% of the total.
- The city must issue a decision within 10 business days of a complete application. Denials must state the reason and the right to appeal to the City Manager within 15 days.

Task types

What this benchmark covers

Completeness review
Checking an application against every item the program's rules require.
Draft permit review
A draft permit checked against the rules and the application, including its numbers.
Deficiency letters
A letter that cites each missing or deficient item to its requirement.
Permit conditions
Conditions drafted from the applicable rules and the facility's application.
Public notices
A notice with the content and timing the program's rules require.
Comment responses
Responses to public comments, grouped by issue and consistent with agency precedent.

Method

How we test

  1. 01

    Real agency deliverables

    One task per deliverable use case: trackers, dashboards, permit systems, forms, decks, workbooks, letters and one-pagers. Each carries its own rules and data, with problems planted where a careful reviewer would catch them.

  2. 02

    Every system can make files

    ChatGPT Astra 6.2, Claude Fable 5.2, Grok 4.7 and Gemini 4 Carbon run on their makers' own services with web search, high reasoning effort and a code tool whose files are what the user receives. When their code crashed or sent no files, they got up to two follow-ups. Justinian ran as shipped, with none.

  3. 03

    Graded as delivered

    Every file is rendered as the requester would open it and every app is opened in a browser. Code or instructions the user would still have to run, deploy or schedule do not count. Bill numbers and rule sections are checked against the official record, and every regulatory call is checked against the statute.

  4. 04

    Three blind judges

    Judges from OpenAI, Anthropic and Google, none of them a competing model, score each deliverable as the user would see it and rank all five systems with names masked. Every system is graded with the same checklist and scoring guide.

Briefing

Test Justinian on your agency's own work.

Request a briefing →

This page’s interactive features did not load. Reloading usually fixes it. You can email sales@vulcan.ai for demos or support@vulcan.ai for help.