All benchmarks
Permits and licensing

An official application built from the rule, with real fillable fields, required attachments and the fee schedule.

Deliverable
A fillable PDF form

Fillable permit application · overall

Deliverable quality score

  • Justinian3.2A-90%
  • ChatGPTAstra 6.2B+82%
  • ClaudeFable 5.2B+82%
  • Grok4.7B80.3%
  • Gemini4 CarbonC+68%

Checklist

What the judges checked

Written before any system ran.

  1. 01A PDF is delivered with real fillable form fields (not just lines to write on)
  2. 02Fields cover every required item in the rules (applicant, event, date/hours, location, attendance, site map attachment, insurance, alcohol license, traffic plan)
  3. 03Yes/no conditions (alcohol, street closure, nonprofit) are checkboxes
  4. 04The fee schedule and a fee worksheet are included and correct
  5. 05Filing deadlines and the decision/appeal timeline are stated correctly
  6. 06Layout is clean and printable on letter paper

Example tasks

What the systems were asked to do

Real tasks from this benchmark. Every system got the same prompt and data.

New applications

Fillable permit application

Create the official fillable PDF application form for this permit, based on the rules below. Use real fillable form fields (text boxes, checkboxes, date…

Full task
Create the official fillable PDF application form for this permit, based on the rules below. Use real fillable form fields (text boxes, checkboxes, date fields) for everything the rules require, include the fee schedule and a fee worksheet, instructions, and a signature and date line.

City of Riverton special event permit rules (Riverton Code ch. 9.40):
- Applies to any event on public property or a public street with 75 or more expected attendees.
- Application must be filed at least 30 days before the event; events over 1,000 attendees need 60 days.
- Required: applicant name and contact, event name, date and hours, location, expected attendance, site map upload, certificate of liability insurance of at least $1,000,000 (at least $2,000,000 if alcohol is served).
- Street closures require a traffic control plan and Public Works sign-off.
- Alcohol requires Police sign-off and proof of a state temporary alcohol license.
- Over 500 attendees requires Fire Marshal sign-off.
- Fees: base $150; plus $2 per expected attendee over 200; street closure $300 per block; alcohol $250. Nonprofit organizations pay 50% of the total.
- The city must issue a decision within 10 business days of a complete application. Denials must state the reason and the right to appeal to the City Manager within 15 days.

Task types

What this benchmark covers

New applications
Every field the rule requires, with validation and instructions.
Renewals and amendments
Forms that carry over prior permit data and ask only what changed.

Method

How we test

  1. 01

    Real agency deliverables

    One task per deliverable use case: trackers, dashboards, permit systems, forms, decks, workbooks, letters and one-pagers. Each carries its own rules and data, with problems planted where a careful reviewer would catch them.

  2. 02

    Every system can make files

    ChatGPT Astra 6.2, Claude Fable 5.2, Grok 4.7 and Gemini 4 Carbon run on their makers' own services with web search, high reasoning effort and a code tool whose files are what the user receives. When their code crashed or sent no files, they got up to two follow-ups. Justinian ran as shipped, with none.

  3. 03

    Graded as delivered

    Every file is rendered as the requester would open it and every app is opened in a browser. Code or instructions the user would still have to run, deploy or schedule do not count. Bill numbers and rule sections are checked against the official record, and every regulatory call is checked against the statute.

  4. 04

    Three blind judges

    Judges from OpenAI, Anthropic and Google, none of them a competing model, score each deliverable as the user would see it and rank all five systems with names masked. Every system is graded with the same checklist and scoring guide.

Briefing

Test Justinian on your agency's own work.

Request a briefing →

This page’s interactive features did not load. Reloading usually fixes it. You can email sales@vulcan.ai for demos or support@vulcan.ai for help.