Public-facing tools that meet Section 508 and WCAG 2.1 AA in the first version delivered.
- Deliverable
- A deployed public-facing app
Accessible app development · overall
Deliverable quality score
- Justinian3.294.7%
- ChatGPTAstra 6.262.7%
- ClaudeFable 5.271.3%
- Grok4.770.7%
- Gemini4 Carbon47%
Checklist
What the judges checked
Written before any system ran.
- 01A public page is delivered that anyone can open by link without signing in
- 02The automated accessibility scan (axe-core, WCAG 2.1 A and AA rules) finds no serious or critical violations on the form and with errors shown (automated)
- 03Every field has a visible label, and required fields and formats are stated in text, not by color alone
- 04Submitting with errors shows an error summary that receives focus and links to each field, and each error appears next to its field and is tied to it for screen readers
- 05Eligibility rules are applied correctly (age 65 from date of birth as of the application date, Medicare number of 11 characters, disability proof types, file type and 5 MB limit, email or phone)
- 06A review screen lets the applicant go back and edit without losing entries, and a confirmation number is shown after confirming
- 07Every control is reachable and usable by keyboard in a logical order, with visible focus (automated keyboard check plus review)
- 08An accessibility note is delivered that says how it was tested and what still needs review
In the field
What agencies did with it
Results agencies reported from their own work in Justinian.
Arkansas Department of Education
PersonnelRebuilding classroom reviews and family intake
- classrooms on one review app
- 4,900
- classrooms on one review app
- hours saved by two tools
- 700
- hours saved by two tools
Example tasks
What the systems were asked to do
Real tasks from this benchmark. Every system got the same prompt and data.
Keyboard and screen reader use
Accessible app development
Build our transit authority's public online application for the Reduced Fare Card. It goes on our public website, so anyone must be able to open it without…
Full task
Build our transit authority's public online application for the Reduced Fare Card. It goes on our public website, so anyone must be able to open it without signing in; give me the link. It must meet WCAG 2.1 Level AA, which is our Section 508 standard. Simulate the submission for now (no real back end): after the applicant confirms, show a confirmation number. Eligibility (the applicant must meet one): age 65 or older on the application date; holds a Medicare card; or has a disability documented by an SSA or SSI disability award letter, a VA disability rating letter, or our Reduced Fare Certification form signed by a licensed health care provider. Fields: first and last name; date of birth; mailing address (the card is mailed); email or phone (at least one is required); which eligibility basis applies; for Medicare, the Medicare number (11 characters); for disability, which proof they have; an upload of the proof document or Medicare card (PDF, JPG or PNG, up to 5 MB); and a checkbox certifying the information is true. Behavior: validate on submit; show an error summary at the top that links to each problem, and show each error next to its field; keep everything the applicant entered; tell applicants under 65 who chose the age basis that they don't qualify on age and can pick another basis; before submitting, show a review screen where they can go back and edit; then show the confirmation number. Also give me a short accessibility note listing how you tested it and anything that still needs review.
Task types
What this benchmark covers
- Keyboard and screen reader use
- Every action reachable and announced without a mouse.
- Color and contrast
- Text and controls that meet the contrast ratios WCAG 2.1 AA sets.
- Forms and errors
- Labeled fields and errors that say what went wrong and how to fix it.
- Documents and media
- Alternatives for images, documents and video.
Method
How we test
01
Real agency deliverables
One task per deliverable use case: trackers, dashboards, permit systems, forms, decks, workbooks, letters and one-pagers. Each carries its own rules and data, with problems planted where a careful reviewer would catch them.
02
Every system can make files
ChatGPT Astra 6.2, Claude Fable 5.2, Grok 4.7 and Gemini 4 Carbon run on their makers' own services with web search, high reasoning effort and a code tool whose files are what the user receives. When their code crashed or sent no files, they got up to two follow-ups. Justinian ran as shipped, with none.
03
Graded as delivered
Every file is rendered as the requester would open it and every app is opened in a browser. Code or instructions the user would still have to run, deploy or schedule do not count. Bill numbers and rule sections are checked against the official record, and every regulatory call is checked against the statute.
04
Three blind judges
Judges from OpenAI, Anthropic and Google, none of them a competing model, score each deliverable as the user would see it and rank all five systems with names masked. Every system is graded with the same checklist and scoring guide.
More in IT and software
All benchmarksBriefing
