A working permit system built from the governing rule: application form, eligibility and completeness checks, fee calculation, reviewer queue and decision notices.
- Deliverable
- A working permit system, run against test applications
Permitting system, end to end · overall
Deliverable quality score
- Justinian3.2C+71.7%
- ChatGPTAstra 6.2C-60.3%
- ClaudeFable 5.2D+53.3%
- Grok4.7D+55.3%
- Gemini4 CarbonC-57%
Checklist
What the judges checked
Written before any system ran.
- 01A working application form is delivered that a person can open and submit
- 02Completeness checks enforce the filing deadlines (30 days, 60 days over 1,000) and required items, including the higher insurance minimum with alcohol
- 03Fees calculate correctly: e.g., a nonprofit, 600 attendees, 2 blocks closed, alcohol: (150 + 800 + 600 + 250) x 50% = $900
- 04Routing sends applications to Public Works (street closure), Police (alcohol) and Fire Marshal (over 500) as the rules require
- 05Decision notices include the reason for denial and the 15-day appeal right to the City Manager
- 06Applications and decisions persist so a reviewer queue actually works across sessions, not just a static mockup
Example tasks
What the systems were asked to do
Real tasks from this benchmark. Every system got the same prompt and data.
Building permits
Permitting system, end to end
Build a working special event permit system for our city from the rules below: an online application form, checks for eligibility and completeness, automatic…
Full task
Build a working special event permit system for our city from the rules below: an online application form, checks for eligibility and completeness, automatic fee calculation, a reviewer queue that routes each application to the departments that must sign off, and decision notices (approval or denial with reason and appeal rights). Share it so anyone with the link can view it, and give me the link. City of Riverton special event permit rules (Riverton Code ch. 9.40): - Applies to any event on public property or a public street with 75 or more expected attendees. - Application must be filed at least 30 days before the event; events over 1,000 attendees need 60 days. - Required: applicant name and contact, event name, date and hours, location, expected attendance, site map upload, certificate of liability insurance of at least $1,000,000 (at least $2,000,000 if alcohol is served). - Street closures require a traffic control plan and Public Works sign-off. - Alcohol requires Police sign-off and proof of a state temporary alcohol license. - Over 500 attendees requires Fire Marshal sign-off. - Fees: base $150; plus $2 per expected attendee over 200; street closure $300 per block; alcohol $250. Nonprofit organizations pay 50% of the total. - The city must issue a decision within 10 business days of a complete application. Denials must state the reason and the right to appeal to the City Manager within 15 days.
Task types
What this benchmark covers
- Building permits
- Residential and commercial permits with plan review, zoning checks and fee schedules.
- Environmental permits
- Air, water and waste permits with completeness rules and public notice.
- Business and professional licenses
- New and renewal licenses with eligibility, reciprocity and continuing education checks.
- Special event and right-of-way permits
- Time-limited permits with insurance, traffic control and sign-off from several departments.
Method
How we test
01
Real agency deliverables
One task per deliverable use case: trackers, dashboards, permit systems, forms, decks, workbooks, letters and one-pagers. Each carries its own rules and data, with problems planted where a careful reviewer would catch them.
02
Every system can make files
ChatGPT Astra 6.2, Claude Fable 5.2, Grok 4.7 and Gemini 4 Carbon run on their makers' own services with web search, high reasoning effort and a code tool whose files are what the user receives. When their code crashed or sent no files, they got up to two follow-ups. Justinian ran as shipped, with none.
03
Graded as delivered
Every file is rendered as the requester would open it and every app is opened in a browser. Code or instructions the user would still have to run, deploy or schedule do not count. Bill numbers and rule sections are checked against the official record, and every regulatory call is checked against the statute.
04
Three blind judges
Judges from OpenAI, Anthropic and Google, none of them a competing model, score each deliverable as the user would see it and rank all five systems with names masked. Every system is graded with the same checklist and scoring guide.
More in Permits and licensing
All benchmarksBriefing
