Code agency IT teams write and maintain: scripts, integrations, security fixes and legacy modernization.
- Deliverable
- Working code with tests
IT coding and automation · overall
Deliverable quality score
- Justinian3.2A+98%
- ChatGPTAstra 6.2A95.7%
- ClaudeFable 5.2A92%
- Grok4.7A+97.3%
- Gemini4 CarbonA92.3%
Checklist
What the judges checked
Written before any system ran.
- 01access_review.py and test_access_review.py are delivered, use only the standard library, and the delivered tests run and pass (automated)
- 02The hidden acceptance tests pass: each rule and edge case on data the system never saw (automated; partial credit by share passed)
- 03The findings for the sample data match the expected findings exactly (automated)
- 04The function and command line match the requested interface (review(hr, ad, as_of) returning account, finding, detail; --as-of and --out)
- 05The script is read-only: it never disables or changes an account
- 06Usage instructions are delivered
- 07A findings CSV and a plain summary for the sample data are delivered and match each other
Example tasks
What the systems were asked to do
Real tasks from this benchmark. Every system got the same prompt and data.
Scripts and automation
IT coding and automation
Our county IT team does a monthly access review by hand. Write a Python script that does it from two CSV exports: the HR roster and the Active Directory…
Full task
Our county IT team does a monthly access review by hand. Write a Python script that does it from two CSV exports: the HR roster and the Active Directory account export (samples below). Our servers have no internet access, so use the Python 3 standard library only, and write the tests with unittest. Requirements: - A module access_review.py with a function review(hr_csv_path, ad_csv_path, as_of) that takes the two file paths and a datetime.date, and returns a list of dicts, one per finding, each with the keys "account" (the sam_account_name), "finding" (one of the codes below) and "detail" (a short plain-English reason). - A command line: python access_review.py HR.csv AD.csv --as-of YYYY-MM-DD --out findings.csv, which writes the findings to a CSV with columns account, finding, detail. - The script only reads the exports. It must never disable or change an account. - Unit tests in test_access_review.py covering each rule and the edge cases, runnable with python -m unittest. Rules (an account can have more than one finding; report each one): - TERMINATED_ENABLED: an enabled account whose employee's HR status is Terminated with a termination date on or before the as-of date. - STALE: an enabled, non-service account whose last logon is more than 90 days before the as-of date. An account that has never logged on (blank last_logon) is stale if it was created more than 90 days before the as-of date. Employees whose HR status is Leave are never stale. - NO_HR_MATCH: an enabled, non-service account that matches no HR record. Match on employee_id; if the account's employee_id is blank, match on email, ignoring case and surrounding spaces. - PRIVILEGED_NON_IT: an enabled account in Domain Admins or Enterprise Admins (group names are case-insensitive) whose matched employee is not in the Information Technology department, or that matches no employee. - SERVICE_NO_OWNER: an enabled service account (sam_account_name starts with svc_, any case) with a blank owner_employee_id, or whose owner is not an Active employee. - Disabled accounts have no findings. Service accounts are never STALE or NO_HR_MATCH. Deliver the script, the tests, short usage instructions, and the findings CSV from running it on the sample data as of November 18, 2026, with a plain summary of what it found. hr_roster.csv employee_id,name,email,department,title,status,termination_date E4001,Hank Diaz,hank.diaz@county.gov,Information Technology,Systems Administrator,Active, E4002,Tara Alvarez,tara.alvarez@county.gov,Public Works,Specialist,Active, E4003,Ivy Brooks,ivy.brooks@county.gov,Information Technology,Help Desk Technician,Active, E4004,Victor Diaz,victor.diaz@county.gov,Finance,Coordinator,Terminated,2026-09-30 E4005,Nate Evans,nate.evans@county.gov,Information Technology,Help Desk Technician,Active, E4006,Luis Jones,luis.jones@county.gov,Planning,Specialist,Active, E4007,Hank Hughes,hank.hughes@county.gov,Finance,Specialist,Active, E4008,Erin Jones,erin.jones@county.gov,Public Works,Coordinator,Terminated,2026-11-18 E4009,Carla Hughes,carla.hughes@county.gov,Human Resources,Specialist,Active, E4010,Erin Evans,erin.evans@county.gov,Finance,Specialist,Active, E4011,Olga Diaz,olga.diaz@county.gov,Public Works,Analyst,Active, E4012,Carla Moore,carla.moore@county.gov,Planning,Manager,Terminated,2026-12-04 E4013,Olga Moore,olga.moore@county.gov,Human Resources,Manager,Active, E4014,Maya Ito,maya.ito@county.gov,Planning,Analyst,Active, E4015,Wes Khan,wes.khan@county.gov,Planning,Specialist,Active, E4016,Zoe Evans,zoe.evans@county.gov,Health,Analyst,Leave, E4017,Luis Moore,luis.moore@county.gov,Finance,Analyst,Active, E4018,Deon Jones,deon.jones@county.gov,Planning,Analyst,Active, E4019,Faisal Hughes,faisal.hughes@county.gov,Planning,Manager,Active, E4020,Maya Lopez,maya.lopez@county.gov,Public Works,Analyst,Terminated,2026-06-15 E4021,Priya Chen,priya.chen@county.gov,Finance,Analyst,Active, E4022,Victor Nguyen,victor.nguyen@county.gov,Public Works,Manager,Active, E4023,Carla Nguyen,carla.nguyen@county.gov,Information Technology,Help Desk Technician,Active, E4024,Carla Chen,carla.chen@county.gov,Public Works,Coordinator,Active, E4025,Sam Ortiz,sam.ortiz@county.gov,Health,Analyst,Active, E4026,Luis Ito,luis.ito@county.gov,Information Technology,Systems Administrator,Active, E4027,Wes Patel,wes.patel@county.gov,Information Technology,Systems Administrator,Active, E4028,Kim Patel,kim.patel@county.gov,Information Technology,Systems Administrator,Active, ad_accounts.csv sam_account_name,employee_id,email,enabled,last_logon,created,groups,owner_employee_id jsmith2,E4999,j.smith@county.gov,TRUE,2026-11-10T09:00:00,2025-01-15,Domain Users, wpatel,E4027,wes.patel@county.gov,TRUE,2026-11-17T12:37:00,2021-03-01,Domain Users, cmoore,E4012,carla.moore@county.gov,TRUE,2026-11-14T13:40:00,2021-03-01,Domain Users, mito,E4014,maya.ito@county.gov,TRUE,2026-11-09T11:48:00,2021-03-01,Domain Users, fhughes,E4019,faisal.hughes@county.gov,TRUE,2026-10-21T12:27:00,2021-03-01,Domain Users, vdiaz,E4004,victor.diaz@county.gov,TRUE,2026-09-29T16:40:00,2021-03-01,Domain Users, wkhan,E4015,wes.khan@county.gov,TRUE,2026-10-19T17:11:00,2021-03-01,Domain Users, tempcontractor,,,TRUE,2026-11-12T13:00:00,2026-08-01,Domain Users;Domain Admins, cchen,E4024,carla.chen@county.gov,TRUE,2026-10-17T17:51:00,2021-03-01,Domain Users, lito,E4026,luis.ito@county.gov,TRUE,2026-11-11T13:29:00,2021-03-01,Domain Users;Enterprise Admins, odiaz,E4011,olga.diaz@county.gov,TRUE,2026-10-16T12:26:00,2021-03-01,Domain Users, svc_backup,,,TRUE,2026-11-18T02:00:00,2020-01-10,Backup Operators,E4001 svc_payroll,,,TRUE,2026-11-15T01:00:00,2021-07-07,Domain Users,E4004 omoore,E4013,olga.moore@county.gov,TRUE,2026-08-19T10:00:00,2021-03-01,Domain Users, sortiz,E4025,sam.ortiz@county.gov,TRUE,,2026-03-02,Domain Users, SVC_Legacy,,,TRUE,2025-01-01T01:00:00,2019-01-01,Domain Users,E4002 zevans,E4016,zoe.evans@county.gov,TRUE,2026-05-02T09:15:00,2021-03-01,Domain Users, hhughes,E4007,hank.hughes@county.gov,TRUE,2026-10-26T16:01:00,2021-03-01,Domain Users, lmoore,E4017,luis.moore@county.gov,TRUE,2026-10-16T16:54:00,2021-03-01,Domain Users;domain admins, chughes,, CARLA.HUGHES@COUNTY.GOV ,TRUE,2026-10-29T09:58:00,2021-03-01,Domain Users, ljones,E4006,luis.jones@county.gov,TRUE,2026-07-01T08:00:00,2021-03-01,Domain Users, eevans,E4010,erin.evans@county.gov,TRUE,2026-08-20T10:00:00,2021-03-01,Domain Users, mlopez,E4020,maya.lopez@county.gov,FALSE,2026-06-14T12:00:00,2021-03-01,Domain Users, kpatel,E4028,kim.patel@county.gov,TRUE,2026-11-13T17:57:00,2021-03-01,Domain Users, ibrooks,E4003,ivy.brooks@county.gov,TRUE,2026-11-17T16:28:00,2021-03-01,Domain Users, hdiaz,E4001,hank.diaz@county.gov,TRUE,2026-11-15T13:48:00,2021-03-01,Domain Users, talvarez,E4002,tara.alvarez@county.gov,TRUE,2026-11-01T14:22:00,2021-03-01,Domain Users, djones,E4018,deon.jones@county.gov,TRUE,2026-10-28T09:52:00,2021-03-01,Domain Users, pchen,E4021,priya.chen@county.gov,TRUE,2026-11-01T08:25:00,2021-03-01,Domain Users, svc_gis_sync,,,TRUE,2026-11-17T03:00:00,2022-05-05,Domain Users, nevans,E4005,nate.evans@county.gov,TRUE,2026-10-30T15:10:00,2021-03-01,Domain Users, cnguyen,E4023,carla.nguyen@county.gov,TRUE,2026-11-09T15:13:00,2021-03-01,Domain Users, vnguyen,E4022,victor.nguyen@county.gov,TRUE,,2026-10-01,Domain Users, oldintern,,intern.2024@county.gov,FALSE,2024-08-09T12:00:00,2024-06-01,Domain Users, ejones,E4008,erin.jones@county.gov,TRUE,2026-10-29T13:08:00,2021-03-01,Domain Users,
Task types
What this benchmark covers
- Scripts and automation
- PowerShell, Python and shell scripts for routine IT administration.
- Integrations and data pipelines
- API connections and scheduled data transfers between agency systems.
- Code review and security fixes
- Vulnerabilities, exposed secrets and unsafe dependencies found and fixed.
- Legacy modernization
- COBOL and other legacy code moved to maintained languages without changing behavior.
- Infrastructure as code
- Cloud configuration that meets the agency security baseline.
Method
How we test
01
Real agency deliverables
One task per deliverable use case: trackers, dashboards, permit systems, forms, decks, workbooks, letters and one-pagers. Each carries its own rules and data, with problems planted where a careful reviewer would catch them.
02
Every system can make files
ChatGPT Astra 6.2, Claude Fable 5.2, Grok 4.7 and Gemini 4 Carbon run on their makers' own services with web search, high reasoning effort and a code tool whose files are what the user receives. When their code crashed or sent no files, they got up to two follow-ups. Justinian ran as shipped, with none.
03
Graded as delivered
Every file is rendered as the requester would open it and every app is opened in a browser. Code or instructions the user would still have to run, deploy or schedule do not count. Bill numbers and rule sections are checked against the official record, and every regulatory call is checked against the statute.
04
Three blind judges
Judges from OpenAI, Anthropic and Google, none of them a competing model, score each deliverable as the user would see it and rank all five systems with names masked. Every system is graded with the same checklist and scoring guide.
More in IT and software
All benchmarksBriefing
