Replies to a batch of constituent emails, each answering the question asked and citing the program rule, routed for approval.
- Deliverable
- Reply drafts and a routing workflow
Constituent responses · overall
Deliverable quality score
- Justinian3.2C+68.7%
- ChatGPTAstra 6.2C-57%
- ClaudeFable 5.2D50.7%
- Grok4.7F41%
- Gemini4 CarbonD+52%
Checklist
What the judges checked
Written before any system ran.
- 01Six reply drafts are delivered, one per email
- 02Each reply answers the question asked correctly under the rules (65+ freeze via application; March purchase cannot claim 2026 since not owner-occupied on Jan 1; 100% DV full exemption; cap applies to taxable not market value and starts second year; renters ineligible; May 15 or 30 days after notice)
- 03Replies are warm, plain-language and ready to send
- 04A routing workflow is actually set up (topic sorting and approval before sending), not only described
- 05Nothing is sent without approval
Example tasks
What the systems were asked to do
Real tasks from this benchmark. Every system got the same prompt and data.
Reply drafts
Constituent responses
I work in a state representative's district office. Draft replies to these six constituent emails, each answering the exact question asked under the rules…
Full task
I work in a state representative's district office. Draft replies to these six constituent emails, each answering the exact question asked under the rules below, in a warm plain-language voice, signed from the office. Then set up a simple routing workflow so future property-tax emails are sorted by topic (exemptions, appraisal caps, protests, other) and sent to me for approval before anything goes out. Set the workflow up; don't just describe it. Emails: 1. "My mom is 68 and her property taxes keep going up. Is there anything that freezes them?" 2. "I bought my house in March 2026. Can I still get the homestead exemption for 2026 or do I wait?" 3. "I'm a disabled veteran rated 100%. Do I owe any property tax on my home?" 4. "Why did my appraisal go up 18% when the law caps it at 10%?" 5. "I rent. Can I claim the homestead exemption?" 6. "I want to protest my value. What's the deadline?" Program rules (state property tax code summary): - Homestead exemption: owner-occupied principal residence on January 1 of the tax year; general residence homestead exemption from school taxes $140,000. Apply by April 30; late applications accepted up to 2 years after the delinquency date. - Owners 65 or older or disabled: additional $60,000 school-tax exemption and a school-tax ceiling (freeze) at the amount paid the year they qualify. Must apply. - 100% disabled veterans (or totally unemployable rating): full exemption on the homestead. - Appraisal cap: a homestead's taxable value cannot rise more than 10% a year; market value can rise more. The cap starts the second year after the owner qualifies. - Renters are not eligible. - Protest deadline: May 15 or 30 days after the appraisal notice was mailed, whichever is later.
Task types
What this benchmark covers
- Reply drafts
- One reply per message, answering what was asked.
- Routing
- Messages sorted by topic and sent to the right reviewer.
Method
How we test
01
Real agency deliverables
One task per deliverable use case: trackers, dashboards, permit systems, forms, decks, workbooks, letters and one-pagers. Each carries its own rules and data, with problems planted where a careful reviewer would catch them.
02
Every system can make files
ChatGPT Astra 6.2, Claude Fable 5.2, Grok 4.7 and Gemini 4 Carbon run on their makers' own services with web search, high reasoning effort and a code tool whose files are what the user receives. When their code crashed or sent no files, they got up to two follow-ups. Justinian ran as shipped, with none.
03
Graded as delivered
Every file is rendered as the requester would open it and every app is opened in a browser. Code or instructions the user would still have to run, deploy or schedule do not count. Bill numbers and rule sections are checked against the official record, and every regulatory call is checked against the statute.
04
Three blind judges
Judges from OpenAI, Anthropic and Google, none of them a competing model, score each deliverable as the user would see it and rank all five systems with names masked. Every system is graded with the same checklist and scoring guide.
More in Workforce and operations
All benchmarksBriefing
