All benchmarks
Grants, contracts and budgets

A solicitation built from the agency's need and its procurement rules, with an evaluation scoring workbook.

Deliverable
A solicitation and a scoring workbook

Solicitation and scoring · overall

Deliverable quality score

  • Justinian3.2A-90.3%
  • ChatGPTAstra 6.2B+86.3%
  • ClaudeFable 5.2B+86.7%
  • Gemini4 CarbonC-58.3%

Checklist

What the judges checked

Written before any system ran.

  1. 01A Word solicitation is delivered with scope, qualifications, instructions, timeline and evaluation criteria
  2. 02It includes Texas-required state contract provisions (for example the Government Code certifications on boycotting Israel, energy companies and firearm entities, the Chapter 2252 disclosure of interested parties, and HUB subcontracting plan requirements)
  3. 03Evaluation criteria weights total 100
  4. 04An Excel scoring workbook is delivered that totals each evaluator's weighted scores and ranks proposals with formulas
  5. 05Terms match the request (210,000 sq ft, 5 nights, 3 years plus two 1-year renewals)
  6. 06Ready to publish with minimal edits

Example tasks

What the systems were asked to do

Real tasks from this benchmark. Every system got the same prompt and data.

Solicitation

Solicitation and scoring

Draft a request for proposals for janitorial services for a Texas state agency's three Austin office buildings (total 210,000 square feet, 5 nights a week,…

Full task
Draft a request for proposals for janitorial services for a Texas state agency's three Austin office buildings (total 210,000 square feet, 5 nights a week, 3-year term with two 1-year renewals). Include the scope of work, minimum qualifications, submission instructions and timeline, the contract terms and certifications Texas state law requires state agency contracts to include, and the evaluation criteria with weights. Also deliver an evaluation scoring workbook for a 3-person committee that totals and ranks proposals automatically.

Task types

What this benchmark covers

Solicitation
Scope, terms and the clauses state law requires.
Evaluation workbook
Weighted criteria and score sheets that total automatically.

Method

How we test

  1. 01

    Real agency deliverables

    One task per deliverable use case: trackers, dashboards, permit systems, forms, decks, workbooks, letters and one-pagers. Each carries its own rules and data, with problems planted where a careful reviewer would catch them.

  2. 02

    Every system can make files

    ChatGPT Astra 6.2, Claude Fable 5.2, Grok 4.7 and Gemini 4 Carbon run on their makers' own services with web search, high reasoning effort and a code tool whose files are what the user receives. When their code crashed or sent no files, they got up to two follow-ups. Justinian ran as shipped, with none.

  3. 03

    Graded as delivered

    Every file is rendered as the requester would open it and every app is opened in a browser. Code or instructions the user would still have to run, deploy or schedule do not count. Bill numbers and rule sections are checked against the official record, and every regulatory call is checked against the statute.

  4. 04

    Three blind judges

    Judges from OpenAI, Anthropic and Google, none of them a competing model, score each deliverable as the user would see it and rank all five systems with names masked. Every system is graded with the same checklist and scoring guide.

Briefing

Test Justinian on your agency's own work.

Request a briefing →

This page’s interactive features did not load. Reloading usually fixes it. You can email sales@vulcan.ai for demos or support@vulcan.ai for help.