All benchmarks
Workforce and operations

Replies to a batch of constituent emails, each answering the question asked and citing the program rule, routed for approval.

Deliverable
Reply drafts and a routing workflow

Constituent responses · overall

Deliverable quality score

  • Justinian3.2C+68.7%
  • ChatGPTAstra 6.2C-57%
  • ClaudeFable 5.2D50.7%
  • Grok4.7F41%
  • Gemini4 CarbonD+52%

Checklist

What the judges checked

Written before any system ran.

  1. 01Six reply drafts are delivered, one per email
  2. 02Each reply answers the question asked correctly under the rules (65+ freeze via application; March purchase cannot claim 2026 since not owner-occupied on Jan 1; 100% DV full exemption; cap applies to taxable not market value and starts second year; renters ineligible; May 15 or 30 days after notice)
  3. 03Replies are warm, plain-language and ready to send
  4. 04A routing workflow is actually set up (topic sorting and approval before sending), not only described
  5. 05Nothing is sent without approval

Example tasks

What the systems were asked to do

Real tasks from this benchmark. Every system got the same prompt and data.

Reply drafts

Constituent responses

I work in a state representative's district office. Draft replies to these six constituent emails, each answering the exact question asked under the rules…

Full task
I work in a state representative's district office. Draft replies to these six constituent emails, each answering the exact question asked under the rules below, in a warm plain-language voice, signed from the office. Then set up a simple routing workflow so future property-tax emails are sorted by topic (exemptions, appraisal caps, protests, other) and sent to me for approval before anything goes out. Set the workflow up; don't just describe it.

Emails:
1. "My mom is 68 and her property taxes keep going up. Is there anything that freezes them?"
2. "I bought my house in March 2026. Can I still get the homestead exemption for 2026 or do I wait?"
3. "I'm a disabled veteran rated 100%. Do I owe any property tax on my home?"
4. "Why did my appraisal go up 18% when the law caps it at 10%?"
5. "I rent. Can I claim the homestead exemption?"
6. "I want to protest my value. What's the deadline?" 

Program rules (state property tax code summary):
- Homestead exemption: owner-occupied principal residence on January 1 of the tax year; general residence homestead exemption from school taxes $140,000. Apply by April 30; late applications accepted up to 2 years after the delinquency date.
- Owners 65 or older or disabled: additional $60,000 school-tax exemption and a school-tax ceiling (freeze) at the amount paid the year they qualify. Must apply.
- 100% disabled veterans (or totally unemployable rating): full exemption on the homestead.
- Appraisal cap: a homestead's taxable value cannot rise more than 10% a year; market value can rise more. The cap starts the second year after the owner qualifies.
- Renters are not eligible.
- Protest deadline: May 15 or 30 days after the appraisal notice was mailed, whichever is later.

Task types

What this benchmark covers

Reply drafts
One reply per message, answering what was asked.
Routing
Messages sorted by topic and sent to the right reviewer.

Method

How we test

  1. 01

    Real agency deliverables

    One task per deliverable use case: trackers, dashboards, permit systems, forms, decks, workbooks, letters and one-pagers. Each carries its own rules and data, with problems planted where a careful reviewer would catch them.

  2. 02

    Every system can make files

    ChatGPT Astra 6.2, Claude Fable 5.2, Grok 4.7 and Gemini 4 Carbon run on their makers' own services with web search, high reasoning effort and a code tool whose files are what the user receives. When their code crashed or sent no files, they got up to two follow-ups. Justinian ran as shipped, with none.

  3. 03

    Graded as delivered

    Every file is rendered as the requester would open it and every app is opened in a browser. Code or instructions the user would still have to run, deploy or schedule do not count. Bill numbers and rule sections are checked against the official record, and every regulatory call is checked against the statute.

  4. 04

    Three blind judges

    Judges from OpenAI, Anthropic and Google, none of them a competing model, score each deliverable as the user would see it and rank all five systems with names masked. Every system is graded with the same checklist and scoring guide.

Briefing

Test Justinian on your agency's own work.

Request a briefing →

This page’s interactive features did not load. Reloading usually fixes it. You can email sales@vulcan.ai for demos or support@vulcan.ai for help.