Completeness reviews, draft permits, deficiency letters, conditions and comment responses, checked against the program's rules.
- Deliverable
- A draft permit document ready for reviewer edits
Permit review and drafting · overall
Deliverable quality score
- Justinian3.2A92.7%
- ChatGPTAstra 6.2C64.3%
- ClaudeFable 5.2A93.3%
- Grok4.7D51.7%
- Gemini4 CarbonC-59%
Checklist
What the judges checked
Written before any system ran.
- 01A Word document is delivered and opens, formatted as a city letter
- 02The decision is a denial (or not-complete notice) because the application was filed fewer than 60 days before an event over 1,000 attendees
- 03It also identifies the insufficient insurance ($1M provided, $2M required with alcohol) and the missing alcohol license
- 04It states the fee correctly if refiled: (150 + 2,000 + 900 + 250) x 50% = $1,650
- 05It states the right to appeal to the City Manager within 15 days
- 06Tone and format are ready to send without edits
Agencies that use it
- Environment, Water & Natural Resources
- Energy & Utilities
- Transportation, Facilities & Infrastructure
In the field
What agencies did with it
Results agencies reported from their own work in Justinian.
Louisiana Department of Environmental Quality
PersonnelReviewing permits faster
- per permit form rules check
- 32 min
- per permit form rules check
- pages of comment responses in 43 minutes
- 1,886
- pages of comment responses in 43 minutes
Example tasks
What the systems were asked to do
Real tasks from this benchmark. Every system got the same prompt and data.
Completeness review
Permit review and drafting
Draft the decision letter (Word document) for this application under the rules below. Application: Riverside Arts Collective (a registered nonprofit), "Night…
Full task
Draft the decision letter (Word document) for this application under the rules below. Application: Riverside Arts Collective (a registered nonprofit), "Night Market on Main," Saturday, December 12, 2026, 4 to 11 p.m., Main Street between 2nd and 5th (3 blocks closed), expected attendance 1,200, beer garden planned. Filed November 18, 2026. Insurance certificate attached: $1,000,000. Traffic control plan attached. State temporary alcohol license: applied for, not yet issued. Decide it correctly under the rules, explain each reason with the code section, state the fee that would apply, and include the appeal rights. City of Riverton special event permit rules (Riverton Code ch. 9.40): - Applies to any event on public property or a public street with 75 or more expected attendees. - Application must be filed at least 30 days before the event; events over 1,000 attendees need 60 days. - Required: applicant name and contact, event name, date and hours, location, expected attendance, site map upload, certificate of liability insurance of at least $1,000,000 (at least $2,000,000 if alcohol is served). - Street closures require a traffic control plan and Public Works sign-off. - Alcohol requires Police sign-off and proof of a state temporary alcohol license. - Over 500 attendees requires Fire Marshal sign-off. - Fees: base $150; plus $2 per expected attendee over 200; street closure $300 per block; alcohol $250. Nonprofit organizations pay 50% of the total. - The city must issue a decision within 10 business days of a complete application. Denials must state the reason and the right to appeal to the City Manager within 15 days.
Task types
What this benchmark covers
- Completeness review
- Checking an application against every item the program's rules require.
- Draft permit review
- A draft permit checked against the rules and the application, including its numbers.
- Deficiency letters
- A letter that cites each missing or deficient item to its requirement.
- Permit conditions
- Conditions drafted from the applicable rules and the facility's application.
- Public notices
- A notice with the content and timing the program's rules require.
- Comment responses
- Responses to public comments, grouped by issue and consistent with agency precedent.
Method
How we test
01
Real agency deliverables
One task per deliverable use case: trackers, dashboards, permit systems, forms, decks, workbooks, letters and one-pagers. Each carries its own rules and data, with problems planted where a careful reviewer would catch them.
02
Every system can make files
ChatGPT Astra 6.2, Claude Fable 5.2, Grok 4.7 and Gemini 4 Carbon run on their makers' own services with web search, high reasoning effort and a code tool whose files are what the user receives. When their code crashed or sent no files, they got up to two follow-ups. Justinian ran as shipped, with none.
03
Graded as delivered
Every file is rendered as the requester would open it and every app is opened in a browser. Code or instructions the user would still have to run, deploy or schedule do not count. Bill numbers and rule sections are checked against the official record, and every regulatory call is checked against the statute.
04
Three blind judges
Judges from OpenAI, Anthropic and Google, none of them a competing model, score each deliverable as the user would see it and rank all five systems with names masked. Every system is graded with the same checklist and scoring guide.
More in Permits and licensing
All benchmarksBriefing
