All benchmarks
Compliance and audit

A sampling workbook with working formulas and a findings memo that ties each exception to its criterion.

Deliverable
A workbook and a findings memo

Audit workpapers · overall

Deliverable quality score

  • Justinian3.2B+83%
  • ChatGPTAstra 6.2A-91.7%
  • ClaudeFable 5.2A93%
  • Grok4.7B+84.3%
  • Gemini4 CarbonC+69.3%

Checklist

What the judges checked

Written before any system ran.

  1. 01An Excel workbook is delivered and the testing uses live formulas, not pasted values
  2. 02All 3 duplicate payments are identified: the same vendor invoice paid twice (P2005/P2100, P2018/P2101, P2034/P2102 by vendor and invoice number)
  3. 03Both payments over $10,000 lacking a second approver are identified (P2110, P2111), and P2112 is correctly not an exception
  4. 04The weekend payment (P2113, Saturday August 22, 2026) is identified
  5. 05No false exceptions are reported
  6. 06A Word memo is delivered with condition, criteria, cause, effect and recommendation for each exception type

Example tasks

What the systems were asked to do

Real tasks from this benchmark. Every system got the same prompt and data.

Sampling and testing

Audit workpapers

I'm testing accounts payable for the July to September 2026 quarter. Policy: (a) the same vendor invoice number may never be paid twice; (b) payments over…

Full task
I'm testing accounts payable for the July to September 2026 quarter. Policy: (a) the same vendor invoice number may never be paid twice; (b) payments over $10,000 need a second approver; (c) payments are only released on business days (Monday to Friday).
Test all 67 payments below. Deliver an Excel workbook with the testing done in live formulas (not pasted values) and an exceptions tab, and a Word findings memo with condition, criteria, cause, effect and recommendation for each exception type.

payment_id,date,vendor,invoice_no,amount,approver_1,approver_2
P2027,2026-09-18,Harbor Print,INV-85992,9774.39,J. Ortiz,
P2056,2026-09-03,Harbor Print,INV-33146,7125.91,J. Ortiz,
P2102,2026-09-24,Acme Supply,INV-20823,9148.55,J. Ortiz,
P2007,2026-08-10,Granite Legal,INV-44961,6934.48,J. Ortiz,
P2004,2026-07-03,Acme Supply,INV-72925,8011.20,J. Ortiz,
P2049,2026-09-07,Acme Supply,INV-67435,1680.83,J. Ortiz,
P2047,2026-09-07,Acme Supply,INV-21650,1919.96,J. Ortiz,
P2059,2026-07-17,Delta Paving,INV-11689,2656.11,J. Ortiz,
P2042,2026-08-27,Granite Legal,INV-19254,6694.68,J. Ortiz,
P2028,2026-08-10,Harbor Print,INV-51994,1389.37,J. Ortiz,
P2043,2026-07-23,Granite Legal,INV-54060,997.36,J. Ortiz,
P2025,2026-09-03,Northside IT,INV-33001,2401.44,J. Ortiz,
P2033,2026-07-31,Acme Supply,INV-80893,974.49,J. Ortiz,
P2008,2026-07-24,Harbor Print,INV-65085,3616.73,J. Ortiz,
P2046,2026-08-05,Acme Supply,INV-70848,7471.09,J. Ortiz,
P2011,2026-09-29,Delta Paving,INV-28329,3617.26,J. Ortiz,
P2010,2026-09-23,Bluebird Catering,INV-34612,5677.29,J. Ortiz,
P2001,2026-08-31,Summit Fuel,INV-61477,2815.92,J. Ortiz,
P2101,2026-10-01,Granite Legal,INV-13066,3429.65,J. Ortiz,
P2041,2026-09-29,Summit Fuel,INV-55262,7400.84,J. Ortiz,
P2015,2026-08-17,Harbor Print,INV-32292,5608.98,J. Ortiz,
P2039,2026-09-28,Granite Legal,INV-28950,3901.68,J. Ortiz,
P2100,2026-07-13,Delta Paving,INV-33345,8744.03,J. Ortiz,
P2052,2026-09-04,Summit Fuel,INV-84653,2031.77,J. Ortiz,
P2053,2026-08-11,Granite Legal,INV-73478,6658.05,J. Ortiz,
P2035,2026-09-07,Acme Supply,INV-14583,8294.32,J. Ortiz,
P2003,2026-07-13,Bluebird Catering,INV-32214,7929.42,J. Ortiz,
P2036,2026-07-24,Acme Supply,INV-11189,4343.18,J. Ortiz,
P2034,2026-09-21,Acme Supply,INV-20823,9148.55,J. Ortiz,
P2113,2026-08-22,Harbor Print,INV-58841,2150.00,J. Ortiz,
P2054,2026-09-28,Northside IT,INV-82100,918.57,J. Ortiz,
P2002,2026-08-19,Granite Legal,INV-35754,7322.55,J. Ortiz,
P2060,2026-09-28,Granite Legal,INV-24440,1584.95,J. Ortiz,
P2026,2026-08-19,Summit Fuel,INV-50993,8585.31,J. Ortiz,
P2040,2026-09-09,Northside IT,INV-76410,2362.12,J. Ortiz,
P2005,2026-07-08,Delta Paving,INV-33345,8744.03,J. Ortiz,
P2016,2026-08-24,Granite Legal,INV-90424,2859.91,J. Ortiz,
P2032,2026-07-10,Bluebird Catering,INV-60079,6147.22,J. Ortiz,
P2044,2026-09-29,Acme Supply,INV-71375,1070.43,J. Ortiz,
P2013,2026-09-28,Granite Legal,INV-65153,8657.42,J. Ortiz,
P2017,2026-07-27,Northside IT,INV-80692,3873.54,J. Ortiz,
P2023,2026-08-24,Delta Paving,INV-99267,4274.66,J. Ortiz,
P2045,2026-08-10,Acme Supply,INV-41274,2909.78,J. Ortiz,
P2009,2026-09-14,Northside IT,INV-86021,544.24,J. Ortiz,
P2058,2026-09-22,Delta Paving,INV-72667,495.79,J. Ortiz,
P2006,2026-08-27,Northside IT,INV-63012,5270.69,J. Ortiz,
P2021,2026-09-14,Summit Fuel,INV-39650,7066.49,J. Ortiz,
P2019,2026-07-14,Granite Legal,INV-39724,7709.98,J. Ortiz,
P2057,2026-09-28,Bluebird Catering,INV-46730,8044.47,J. Ortiz,
P2111,2026-09-03,Summit Fuel,INV-60977,11980.40,J. Ortiz,
P2037,2026-07-20,Granite Legal,INV-66693,7695.85,J. Ortiz,
P2030,2026-08-20,Acme Supply,INV-80190,1947.05,J. Ortiz,
P2050,2026-07-16,Northside IT,INV-31244,366.29,J. Ortiz,
P2055,2026-08-03,Acme Supply,INV-50085,8045.55,J. Ortiz,
P2029,2026-07-22,Acme Supply,INV-96946,7379.63,J. Ortiz,
P2110,2026-08-14,Delta Paving,INV-55102,14250.00,J. Ortiz,
P2018,2026-09-28,Granite Legal,INV-13066,3429.65,J. Ortiz,
P2038,2026-08-07,Acme Supply,INV-83749,8950.49,J. Ortiz,
P2024,2026-09-21,Bluebird Catering,INV-47449,8691.33,J. Ortiz,
P2048,2026-07-24,Harbor Print,INV-63999,4717.75,J. Ortiz,
P2051,2026-07-06,Harbor Print,INV-99369,2563.51,J. Ortiz,
P2031,2026-08-12,Delta Paving,INV-41277,3928.44,J. Ortiz,
P2014,2026-09-17,Acme Supply,INV-29311,6379.55,J. Ortiz,
P2020,2026-09-11,Bluebird Catering,INV-37691,9138.24,J. Ortiz,
P2012,2026-07-20,Granite Legal,INV-90680,4203.79,J. Ortiz,
P2022,2026-07-16,Summit Fuel,INV-77177,7364.34,J. Ortiz,
P2112,2026-09-10,Northside IT,INV-61210,12400.00,J. Ortiz,M. Chen

Task types

What this benchmark covers

Sampling and testing
Samples drawn and tested in a workbook with live formulas.
Findings memo
Condition, criterion, cause and effect for each exception.

Method

How we test

  1. 01

    Real agency deliverables

    One task per deliverable use case: trackers, dashboards, permit systems, forms, decks, workbooks, letters and one-pagers. Each carries its own rules and data, with problems planted where a careful reviewer would catch them.

  2. 02

    Every system can make files

    ChatGPT Astra 6.2, Claude Fable 5.2, Grok 4.7 and Gemini 4 Carbon run on their makers' own services with web search, high reasoning effort and a code tool whose files are what the user receives. When their code crashed or sent no files, they got up to two follow-ups. Justinian ran as shipped, with none.

  3. 03

    Graded as delivered

    Every file is rendered as the requester would open it and every app is opened in a browser. Code or instructions the user would still have to run, deploy or schedule do not count. Bill numbers and rule sections are checked against the official record, and every regulatory call is checked against the statute.

  4. 04

    Three blind judges

    Judges from OpenAI, Anthropic and Google, none of them a competing model, score each deliverable as the user would see it and rank all five systems with names masked. Every system is graded with the same checklist and scoring guide.

More in Compliance and audit

All benchmarks

Briefing

Test Justinian on your agency's own work.

Request a briefing →

This page’s interactive features did not load. Reloading usually fixes it. You can email sales@vulcan.ai for demos or support@vulcan.ai for help.