All benchmarks
IT and software

Access control, audit logging, document handling and internal tools for FedRAMP Moderate clouds, scored on whether the code works and on the controls a security review checks.

Deliverable
A deployed internal agency app

Secure software development · overall

Deliverable quality score

  • Justinian3.2
    74.3%
  • ChatGPTAstra 6.2
    57%
  • ClaudeFable 5.2
    57.7%

Checklist

What the judges checked

Written before any system ran.

  1. 01A working app is delivered that a person can open, behind sign-in as requested
  2. 02Sign-in uses individual accounts through a real identity provider or platform sign-in, with no shared or hard-coded passwords and no way for a user to pick their own role (automated security review)
  3. 03The three roles are enforced where the data is stored or served, not only by hiding buttons in browser code, and match the requested permissions (automated security review)
  4. 04Data and the audit log are stored centrally so every user sees the same records, not in one browser's local storage (automated security review)
  5. 05The audit log records who, what, which tip and when for creates, assignments, status changes and exports, and no role can edit or delete it (automated security review)
  6. 06Narratives and other user input are handled as untrusted text: no script injection, no secrets in client code (automated security review)
  7. 07A Word control mapping is delivered whose NIST SP 800-53 Rev. 5 control IDs are real and correctly matched to the features
  8. 08The source code is delivered as files

Example tasks

What the systems were asked to do

Real tasks from this benchmark. Every system got the same prompt and data.

Internal tools and dashboards

Secure software development

I'm on the IT team at a state Office of Inspector General. Build and deploy an internal web app for our fraud hotline tips. It will hold sensitive allegations…

Full task
I'm on the IT team at a state Office of Inspector General. Build and deploy an internal web app for our fraud hotline tips. It will hold sensitive allegations and names, and our security team will review it against NIST SP 800-53 Rev. 5 (moderate baseline) before we use it.
Requirements:
- Only people we add can get in, and they sign in with their own accounts. No shared passwords, and nobody picks their own role.
- Roles: Intake staff (log new tips and see only the tips they logged), Investigator (see and update only the tips assigned to them), Supervisor (see all tips, assign tips to investigators, export a CSV, and manage who has which role).
- Tip fields: tip ID (assigned automatically), date received, source (hotline, email, web form, referral), allegation category, agency involved, subject name, narrative, status (New, Assigned, Under review, Closed) and assigned investigator.
- Every tip created, assignment, status change and export is recorded in an audit log showing who did it, what changed, which tip and when. No one, including Supervisors, can edit or delete the audit log.
- Tip narratives come from the public and can contain anything; the app must treat them as untrusted text.
I'm the first Supervisor; I'll add the staff myself once it's built.
Deliver (1) the working app and its link, (2) a short Word document mapping each security feature to the NIST SP 800-53 Rev. 5 control it implements, and (3) the source code as files for our security review.

Task types

What this benchmark covers

Internal tools and dashboards2 tasks
A working tool built from the agency's stated requirements.
Access control1 task
Roles and permissions that match who should see and change what.
Audit logging1 task
A record of who did what and when, as the control baseline requires.
Data handling1 task
Encryption, retention and boundaries for the data the app touches.

Method

How we test

  1. 01

    Real agency deliverables

    One task per deliverable use case: trackers, dashboards, permit systems, forms, decks, workbooks, letters and one-pagers. Each carries its own rules and data, with problems planted where a careful reviewer would catch them.

  2. 02

    Every system can make files

    ChatGPT Astra 6.2, Claude Fable 5.2, Grok 4.7 and Gemini 4 Carbon run on their makers' own services with web search, high reasoning effort and a code tool whose files are what the user receives. When their code crashed or sent no files, they got up to two follow-ups. Justinian ran as shipped, with none.

  3. 03

    Graded as delivered

    Every file is rendered as the requester would open it and every app is opened in a browser. Code or instructions the user would still have to run, deploy or schedule do not count. Bill numbers and rule sections are checked against the official record, and every regulatory call is checked against the statute.

  4. 04

    Three blind judges

    Judges from OpenAI, Anthropic and Google, none of them a competing model, score each deliverable as the user would see it and rank all five systems with names masked. Every system is graded with the same checklist and scoring guide.

Briefing

Test Justinian on your agency's own work.

Request a briefing →

This page’s interactive features did not load. Reloading usually fixes it. You can email sales@vulcan.ai for demos or support@vulcan.ai for help.