An official application built from the rule, with real fillable fields, required attachments and the fee schedule.
- Deliverable
- A fillable PDF form
Fillable permit application · overall
Deliverable quality score
- Justinian3.2A-90%
- ChatGPTAstra 6.2B+82%
- ClaudeFable 5.2B+82%
- Grok4.7B80.3%
- Gemini4 CarbonC+68%
Checklist
What the judges checked
Written before any system ran.
- 01A PDF is delivered with real fillable form fields (not just lines to write on)
- 02Fields cover every required item in the rules (applicant, event, date/hours, location, attendance, site map attachment, insurance, alcohol license, traffic plan)
- 03Yes/no conditions (alcohol, street closure, nonprofit) are checkboxes
- 04The fee schedule and a fee worksheet are included and correct
- 05Filing deadlines and the decision/appeal timeline are stated correctly
- 06Layout is clean and printable on letter paper
Example tasks
What the systems were asked to do
Real tasks from this benchmark. Every system got the same prompt and data.
New applications
Fillable permit application
Create the official fillable PDF application form for this permit, based on the rules below. Use real fillable form fields (text boxes, checkboxes, date…
Full task
Create the official fillable PDF application form for this permit, based on the rules below. Use real fillable form fields (text boxes, checkboxes, date fields) for everything the rules require, include the fee schedule and a fee worksheet, instructions, and a signature and date line. City of Riverton special event permit rules (Riverton Code ch. 9.40): - Applies to any event on public property or a public street with 75 or more expected attendees. - Application must be filed at least 30 days before the event; events over 1,000 attendees need 60 days. - Required: applicant name and contact, event name, date and hours, location, expected attendance, site map upload, certificate of liability insurance of at least $1,000,000 (at least $2,000,000 if alcohol is served). - Street closures require a traffic control plan and Public Works sign-off. - Alcohol requires Police sign-off and proof of a state temporary alcohol license. - Over 500 attendees requires Fire Marshal sign-off. - Fees: base $150; plus $2 per expected attendee over 200; street closure $300 per block; alcohol $250. Nonprofit organizations pay 50% of the total. - The city must issue a decision within 10 business days of a complete application. Denials must state the reason and the right to appeal to the City Manager within 15 days.
Task types
What this benchmark covers
- New applications
- Every field the rule requires, with validation and instructions.
- Renewals and amendments
- Forms that carry over prior permit data and ask only what changed.
Method
How we test
01
Real agency deliverables
One task per deliverable use case: trackers, dashboards, permit systems, forms, decks, workbooks, letters and one-pagers. Each carries its own rules and data, with problems planted where a careful reviewer would catch them.
02
Every system can make files
ChatGPT Astra 6.2, Claude Fable 5.2, Grok 4.7 and Gemini 4 Carbon run on their makers' own services with web search, high reasoning effort and a code tool whose files are what the user receives. When their code crashed or sent no files, they got up to two follow-ups. Justinian ran as shipped, with none.
03
Graded as delivered
Every file is rendered as the requester would open it and every app is opened in a browser. Code or instructions the user would still have to run, deploy or schedule do not count. Bill numbers and rule sections are checked against the official record, and every regulatory call is checked against the statute.
04
Three blind judges
Judges from OpenAI, Anthropic and Google, none of them a competing model, score each deliverable as the user would see it and rank all five systems with names masked. Every system is graded with the same checklist and scoring guide.
More in Permits and licensing
All benchmarksBriefing
