All benchmarks
IT and software

Code agency IT teams write and maintain: scripts, integrations, security fixes and legacy modernization.

Deliverable
Working code with tests

IT coding and automation · overall

Deliverable quality score

  • Justinian3.2A+98%
  • ChatGPTAstra 6.2A95.7%
  • ClaudeFable 5.2A92%
  • Grok4.7A+97.3%
  • Gemini4 CarbonA92.3%

Checklist

What the judges checked

Written before any system ran.

  1. 01access_review.py and test_access_review.py are delivered, use only the standard library, and the delivered tests run and pass (automated)
  2. 02The hidden acceptance tests pass: each rule and edge case on data the system never saw (automated; partial credit by share passed)
  3. 03The findings for the sample data match the expected findings exactly (automated)
  4. 04The function and command line match the requested interface (review(hr, ad, as_of) returning account, finding, detail; --as-of and --out)
  5. 05The script is read-only: it never disables or changes an account
  6. 06Usage instructions are delivered
  7. 07A findings CSV and a plain summary for the sample data are delivered and match each other

Example tasks

What the systems were asked to do

Real tasks from this benchmark. Every system got the same prompt and data.

Scripts and automation

IT coding and automation

Our county IT team does a monthly access review by hand. Write a Python script that does it from two CSV exports: the HR roster and the Active Directory…

Full task
Our county IT team does a monthly access review by hand. Write a Python script that does it from two CSV exports: the HR roster and the Active Directory account export (samples below). Our servers have no internet access, so use the Python 3 standard library only, and write the tests with unittest.
Requirements:
- A module access_review.py with a function review(hr_csv_path, ad_csv_path, as_of) that takes the two file paths and a datetime.date, and returns a list of dicts, one per finding, each with the keys "account" (the sam_account_name), "finding" (one of the codes below) and "detail" (a short plain-English reason).
- A command line: python access_review.py HR.csv AD.csv --as-of YYYY-MM-DD --out findings.csv, which writes the findings to a CSV with columns account, finding, detail.
- The script only reads the exports. It must never disable or change an account.
- Unit tests in test_access_review.py covering each rule and the edge cases, runnable with python -m unittest.
Rules (an account can have more than one finding; report each one):
- TERMINATED_ENABLED: an enabled account whose employee's HR status is Terminated with a termination date on or before the as-of date.
- STALE: an enabled, non-service account whose last logon is more than 90 days before the as-of date. An account that has never logged on (blank last_logon) is stale if it was created more than 90 days before the as-of date. Employees whose HR status is Leave are never stale.
- NO_HR_MATCH: an enabled, non-service account that matches no HR record. Match on employee_id; if the account's employee_id is blank, match on email, ignoring case and surrounding spaces.
- PRIVILEGED_NON_IT: an enabled account in Domain Admins or Enterprise Admins (group names are case-insensitive) whose matched employee is not in the Information Technology department, or that matches no employee.
- SERVICE_NO_OWNER: an enabled service account (sam_account_name starts with svc_, any case) with a blank owner_employee_id, or whose owner is not an Active employee.
- Disabled accounts have no findings. Service accounts are never STALE or NO_HR_MATCH.
Deliver the script, the tests, short usage instructions, and the findings CSV from running it on the sample data as of November 18, 2026, with a plain summary of what it found.

hr_roster.csv
employee_id,name,email,department,title,status,termination_date
E4001,Hank Diaz,hank.diaz@county.gov,Information Technology,Systems Administrator,Active,
E4002,Tara Alvarez,tara.alvarez@county.gov,Public Works,Specialist,Active,
E4003,Ivy Brooks,ivy.brooks@county.gov,Information Technology,Help Desk Technician,Active,
E4004,Victor Diaz,victor.diaz@county.gov,Finance,Coordinator,Terminated,2026-09-30
E4005,Nate Evans,nate.evans@county.gov,Information Technology,Help Desk Technician,Active,
E4006,Luis Jones,luis.jones@county.gov,Planning,Specialist,Active,
E4007,Hank Hughes,hank.hughes@county.gov,Finance,Specialist,Active,
E4008,Erin Jones,erin.jones@county.gov,Public Works,Coordinator,Terminated,2026-11-18
E4009,Carla Hughes,carla.hughes@county.gov,Human Resources,Specialist,Active,
E4010,Erin Evans,erin.evans@county.gov,Finance,Specialist,Active,
E4011,Olga Diaz,olga.diaz@county.gov,Public Works,Analyst,Active,
E4012,Carla Moore,carla.moore@county.gov,Planning,Manager,Terminated,2026-12-04
E4013,Olga Moore,olga.moore@county.gov,Human Resources,Manager,Active,
E4014,Maya Ito,maya.ito@county.gov,Planning,Analyst,Active,
E4015,Wes Khan,wes.khan@county.gov,Planning,Specialist,Active,
E4016,Zoe Evans,zoe.evans@county.gov,Health,Analyst,Leave,
E4017,Luis Moore,luis.moore@county.gov,Finance,Analyst,Active,
E4018,Deon Jones,deon.jones@county.gov,Planning,Analyst,Active,
E4019,Faisal Hughes,faisal.hughes@county.gov,Planning,Manager,Active,
E4020,Maya Lopez,maya.lopez@county.gov,Public Works,Analyst,Terminated,2026-06-15
E4021,Priya Chen,priya.chen@county.gov,Finance,Analyst,Active,
E4022,Victor Nguyen,victor.nguyen@county.gov,Public Works,Manager,Active,
E4023,Carla Nguyen,carla.nguyen@county.gov,Information Technology,Help Desk Technician,Active,
E4024,Carla Chen,carla.chen@county.gov,Public Works,Coordinator,Active,
E4025,Sam Ortiz,sam.ortiz@county.gov,Health,Analyst,Active,
E4026,Luis Ito,luis.ito@county.gov,Information Technology,Systems Administrator,Active,
E4027,Wes Patel,wes.patel@county.gov,Information Technology,Systems Administrator,Active,
E4028,Kim Patel,kim.patel@county.gov,Information Technology,Systems Administrator,Active,

ad_accounts.csv
sam_account_name,employee_id,email,enabled,last_logon,created,groups,owner_employee_id
jsmith2,E4999,j.smith@county.gov,TRUE,2026-11-10T09:00:00,2025-01-15,Domain Users,
wpatel,E4027,wes.patel@county.gov,TRUE,2026-11-17T12:37:00,2021-03-01,Domain Users,
cmoore,E4012,carla.moore@county.gov,TRUE,2026-11-14T13:40:00,2021-03-01,Domain Users,
mito,E4014,maya.ito@county.gov,TRUE,2026-11-09T11:48:00,2021-03-01,Domain Users,
fhughes,E4019,faisal.hughes@county.gov,TRUE,2026-10-21T12:27:00,2021-03-01,Domain Users,
vdiaz,E4004,victor.diaz@county.gov,TRUE,2026-09-29T16:40:00,2021-03-01,Domain Users,
wkhan,E4015,wes.khan@county.gov,TRUE,2026-10-19T17:11:00,2021-03-01,Domain Users,
tempcontractor,,,TRUE,2026-11-12T13:00:00,2026-08-01,Domain Users;Domain Admins,
cchen,E4024,carla.chen@county.gov,TRUE,2026-10-17T17:51:00,2021-03-01,Domain Users,
lito,E4026,luis.ito@county.gov,TRUE,2026-11-11T13:29:00,2021-03-01,Domain Users;Enterprise Admins,
odiaz,E4011,olga.diaz@county.gov,TRUE,2026-10-16T12:26:00,2021-03-01,Domain Users,
svc_backup,,,TRUE,2026-11-18T02:00:00,2020-01-10,Backup Operators,E4001
svc_payroll,,,TRUE,2026-11-15T01:00:00,2021-07-07,Domain Users,E4004
omoore,E4013,olga.moore@county.gov,TRUE,2026-08-19T10:00:00,2021-03-01,Domain Users,
sortiz,E4025,sam.ortiz@county.gov,TRUE,,2026-03-02,Domain Users,
SVC_Legacy,,,TRUE,2025-01-01T01:00:00,2019-01-01,Domain Users,E4002
zevans,E4016,zoe.evans@county.gov,TRUE,2026-05-02T09:15:00,2021-03-01,Domain Users,
hhughes,E4007,hank.hughes@county.gov,TRUE,2026-10-26T16:01:00,2021-03-01,Domain Users,
lmoore,E4017,luis.moore@county.gov,TRUE,2026-10-16T16:54:00,2021-03-01,Domain Users;domain admins,
chughes,,  CARLA.HUGHES@COUNTY.GOV ,TRUE,2026-10-29T09:58:00,2021-03-01,Domain Users,
ljones,E4006,luis.jones@county.gov,TRUE,2026-07-01T08:00:00,2021-03-01,Domain Users,
eevans,E4010,erin.evans@county.gov,TRUE,2026-08-20T10:00:00,2021-03-01,Domain Users,
mlopez,E4020,maya.lopez@county.gov,FALSE,2026-06-14T12:00:00,2021-03-01,Domain Users,
kpatel,E4028,kim.patel@county.gov,TRUE,2026-11-13T17:57:00,2021-03-01,Domain Users,
ibrooks,E4003,ivy.brooks@county.gov,TRUE,2026-11-17T16:28:00,2021-03-01,Domain Users,
hdiaz,E4001,hank.diaz@county.gov,TRUE,2026-11-15T13:48:00,2021-03-01,Domain Users,
talvarez,E4002,tara.alvarez@county.gov,TRUE,2026-11-01T14:22:00,2021-03-01,Domain Users,
djones,E4018,deon.jones@county.gov,TRUE,2026-10-28T09:52:00,2021-03-01,Domain Users,
pchen,E4021,priya.chen@county.gov,TRUE,2026-11-01T08:25:00,2021-03-01,Domain Users,
svc_gis_sync,,,TRUE,2026-11-17T03:00:00,2022-05-05,Domain Users,
nevans,E4005,nate.evans@county.gov,TRUE,2026-10-30T15:10:00,2021-03-01,Domain Users,
cnguyen,E4023,carla.nguyen@county.gov,TRUE,2026-11-09T15:13:00,2021-03-01,Domain Users,
vnguyen,E4022,victor.nguyen@county.gov,TRUE,,2026-10-01,Domain Users,
oldintern,,intern.2024@county.gov,FALSE,2024-08-09T12:00:00,2024-06-01,Domain Users,
ejones,E4008,erin.jones@county.gov,TRUE,2026-10-29T13:08:00,2021-03-01,Domain Users,

Task types

What this benchmark covers

Scripts and automation
PowerShell, Python and shell scripts for routine IT administration.
Integrations and data pipelines
API connections and scheduled data transfers between agency systems.
Code review and security fixes
Vulnerabilities, exposed secrets and unsafe dependencies found and fixed.
Legacy modernization
COBOL and other legacy code moved to maintained languages without changing behavior.
Infrastructure as code
Cloud configuration that meets the agency security baseline.

Method

How we test

  1. 01

    Real agency deliverables

    One task per deliverable use case: trackers, dashboards, permit systems, forms, decks, workbooks, letters and one-pagers. Each carries its own rules and data, with problems planted where a careful reviewer would catch them.

  2. 02

    Every system can make files

    ChatGPT Astra 6.2, Claude Fable 5.2, Grok 4.7 and Gemini 4 Carbon run on their makers' own services with web search, high reasoning effort and a code tool whose files are what the user receives. When their code crashed or sent no files, they got up to two follow-ups. Justinian ran as shipped, with none.

  3. 03

    Graded as delivered

    Every file is rendered as the requester would open it and every app is opened in a browser. Code or instructions the user would still have to run, deploy or schedule do not count. Bill numbers and rule sections are checked against the official record, and every regulatory call is checked against the statute.

  4. 04

    Three blind judges

    Judges from OpenAI, Anthropic and Google, none of them a competing model, score each deliverable as the user would see it and rank all five systems with names masked. Every system is graded with the same checklist and scoring guide.

Briefing

Test Justinian on your agency's own work.

Request a briefing →

This page’s interactive features did not load. Reloading usually fixes it. You can email sales@vulcan.ai for demos or support@vulcan.ai for help.