All benchmarks
Data and maps

Agency records joined to boundaries, facilities and permits, and turned into a map staff can share.

Deliverable
A working, shareable map

GIS mapping · overall

Deliverable quality score

  • Justinian3.2A94.7%
  • ChatGPTAstra 6.2B+85.3%
  • ClaudeFable 5.2B78%
  • Grok4.7B+85.7%
  • Gemini4 CarbonF20.3%

Checklist

What the judges checked

Written before any system ran.

  1. 01A map is delivered that a person can open, with all 30 facilities plotted at their coordinates
  2. 02FEMA's flood hazard zones from the National Flood Hazard Layer are shown on the map, at least the Special Flood Hazard Area
  3. 03The city limits of Baton Rouge, Baker, Central and Zachary are shown on the map
  4. 04Each facility's Special Flood Hazard Area status matches FEMA's National Flood Hazard Layer (10 of 30 are in it: CARRIAGEWOOD ESTATES, COPPER MILL GOLF COMMUNITY, EXCLUSIVE HOMES BY CHUCK WALTERS LLC, M&J CIVIL CONSTRUCTION LLC - SPECIALTY SURGICAL HOSPITAL, OAK CLUSTER SUBDIVISION, SANTA MARIA, ST. PIERRE PROPERTY, TJC BUILDERS LLC- THE SETTLEMENT @ WILLOW GROVE, TRACY JORDAN LLC, UNIFIED CONSTRUCTION GROUP LLC) (automated check of the spreadsheet)
  5. 05Each facility's flood zone and zone subtype match the National Flood Hazard Layer (X (AREA OF MINIMAL FLOOD HAZARD) 10; X (AREA WITH REDUCED FLOOD RISK DUE TO LEVEE) 9; AE 9; A 1; X (0.2 PCT ANNUAL CHANCE FLOOD HAZARD) 1) (automated check of the spreadsheet)
  6. 06Each facility's jurisdiction matches the Census 2025 city boundaries (all 30: Baton Rouge 11; Central 5; Zachary 3; St. George 6; Unincorporated parish 5). Key correction before grading: the request listed four cities, but St. George (incorporated 2024) is also a city in the Census 2025 boundaries, and 6 facilities are inside it (automated check of the spreadsheet)
  7. 07The summary table of facilities in the Special Flood Hazard Area by jurisdiction is correct (Baton Rouge 2 of 11; Central 3 of 5; Zachary 1 of 3; St. George 4 of 6; Unincorporated parish 0 of 5)
  8. 08Facilities in the Special Flood Hazard Area are clearly marked, each facility has a popup with its name, flood zone and jurisdiction, and the map is clear enough for staff to use without training

Agencies that use it

  • Transportation, Facilities & Infrastructure

In the field

What agencies did with it

Results agencies reported from their own work in Justinian.

Arkansas Department of Energy and Environment

Consulting

Managing agency data in-house

database migration quote avoided
$150K
database migration quote avoided
GIS analyst position no longer needed
$77K/yr
GIS analyst position no longer needed

Example tasks

What the systems were asked to do

Real tasks from this benchmark. Every system got the same prompt and data.

Record-to-boundary joins

GIS mapping

I'm with the East Baton Rouge Parish floodplain management office. We're checking which EPA-permitted wastewater and stormwater (NPDES) facilities in the…

Full task
I'm with the East Baton Rouge Parish floodplain management office. We're checking which EPA-permitted wastewater and stormwater (NPDES) facilities in the parish sit in FEMA's Special Flood Hazard Area, and whose floodplain rules apply to each one. The facility list below is from EPA's ECHO database (FRS registry ID, address and EPA's coordinates).
For each facility, tell me its current FEMA flood zone from FEMA's National Flood Hazard Layer (for zone X, include the subtype, such as reduced risk due to levee or 0.2 percent annual chance), whether it is in the Special Flood Hazard Area, and whether it is inside the city limits of Baton Rouge, Baker, Central or Zachary or in the unincorporated parish, using the Census Bureau's 2025 city boundaries.
Deliver:
1. A map our staff can open that shows FEMA's flood hazard zones, the four city limits, and the facilities, with facilities in the Special Flood Hazard Area clearly marked and a popup for each facility with its name, flood zone and jurisdiction. Share it so anyone with the link can view it, and give me the link.
2. A summary table of how many facilities in each jurisdiction are in the Special Flood Hazard Area.
3. A spreadsheet with one row per facility: registry ID, name, flood zone, zone subtype, in the Special Flood Hazard Area (yes or no) and jurisdiction.

frs_registry_id,facility_name,street_address,mailing_city,zip,latitude,longitude
110011146937,"BREC'S BATON ROUGE ZOO","3601 THOMAS ROAD",BATON ROUGE,70807,30.563280,-91.158450
110037932146,"THE OAKS","TBD HIGHLAND OAKS DR.",ZACHARY,70000,30.662578,-91.158133
110056151122,"NOTO DEVELOPMENT LLC - UNIVERSITY CLUB PLANTATIONS","11959 NICHOLSON DR",BATON ROUGE,70810,30.345160,-91.139970
110041910260,"COPPER MILL GOLF COMMUNITY","2438 CARNOUSTIE DR.",ZACHARY,70791,30.634540,-91.182630
110071545181,"M&J CIVIL CONSTRUCTION LLC - SPECIALTY SURGICAL HOSPITAL","6330 MOURNING DOVE DR",BATON ROUGE,70808,30.392083,-91.037806
110071729438,"BLUE SKY PROPERTIES JLU, LLC - LION'S LAWN CARE & LANDSCAPE LLC","430 SOUTH STEVENDALE RD",BATON ROUGE,70819,30.463300,-91.003620
110038516629,"OAK CLUSTER SUBDIVISION","6442 OAK CLUSTER DR.",CENTRAL,70739,30.509810,-91.009760
110070235173,"TJC BUILDERS LLC- THE SETTLEMENT @ WILLOW GROVE","OFF PERKINS RD",BATON ROUGE,70817,30.375109,-90.980419
110071896602,"DOLLAR TREE STORES, INC. - FAMILY DOLLAR #28831","801 O'NEAL LANE",BATON ROUGE,70808,30.457220,-91.008410
110054107195,"THE SETTLEMENT AT WILLOW GROVE","BLUEBONNET BLVD & PERKINS RD",BATON ROUGE,70808,30.379250,-91.097970
110040438559,"TRACY JORDAN LLC","3549 FAIRWOOD",BATON ROUGE,70805,30.481210,-91.141660
110030757704,"BLD BUILDERS, INC.","AMBER LAKES SUBDIVISION",GREENWELL SPRINGS,70739,30.567770,-91.002990
110042382579,"SANTA MARIA","19634 FAIRWAY OAKS AVE.",BATON ROUGE,70817,30.337820,-91.001930
110030997205,"MENZIE STONE COMPANY, INC.","11145 DARRYL DRIVE",BATON ROUGE,70815,30.471010,-91.059890
110072201866,"ENGQUIST-ROUZAN COMMERCIAL DEVELOPMENT LLC - ROUZAN","SE CORNER OF PERKINS RD & GALSGOW AVE.",BATON ROUGE,70806,30.448797,-91.124704
110011176066,"COUNTRY AIRE MOBILE HOME PARK","21020 HWY 19",ZACHARY,70791,30.649627,-91.157635
110044892621,"GREENTRAILS","15554 PARKSIDE COURT",BATON ROUGE,70808,30.394540,-91.010860
110070626005,"THE LEMOINE COMPANY, LLC - GEO NEXT GENERATION HIGH SCHOOL","2121 N SHERWOOD FOREST BLVD",BATON ROUGE,70817,30.468180,-91.058090
110055631534,"UNIFIED CONSTRUCTION GROUP LLC","2136 SUGAR CANE LN",BATON ROUGE,70000,30.454590,-91.069520
110041247610,"DEMCO/THOMPSON CREEK SUBDIVISION","22520 SAMUELS RD (US HWY 61)",ZACHARY,70791,30.688281,-91.267845
110043742073,"ST. PIERRE PROPERTY","LIBERY & ALPHONSE FORBES RD",PRIDE,70000,30.597070,-91.020809
110022464135,"CVS PHARMACY, INC.","14190 GREENWELL SPRINGS RD",GREENWELL SPRINGS,70739,30.511660,-91.029710
110012671901,"EXTREME CLEAN CAR WASH","LEE DRIVE LLC",BATON ROUGE,70808,30.397136,-91.156217
110071444171,"ARKEL CONSTRUCTION LLC - ALL STAR AUTOMOTIVE USED CARS","10313 AIRLINE HWY",BATON ROUGE,70802,30.416820,-91.070020
110069423877,"EXCLUSIVE HOMES BY CHUCK WALTERS LLC","THE VILLAGE AT MAGNOLIA SQUARE",CENTRAL,70000,30.559305,-91.035507
110032945829,"ST.LOUIS POINTE RESIDECES LP","STAN AVE. &",BATON ROUGE,70805,30.470090,-91.054430
110070050269,"CIRCLE K STORES INC.","22015 S SHERWOOD FOREST BLVD",BATON ROUGE,70808,30.432500,-91.056944
110043729132,"CARRIAGEWOOD ESTATES","18515 SURREY COURT AVE. (LOT 39)",BATON ROUGE,70808,30.372040,-90.982380
110064429523,"THE HAMILTON FAMILY LLC","6575 AIRLINE HWY",BATON ROUGE,70808,30.492981,-91.124444
110070112331,"PLAIN CONSTRUCTION LLC - CARRIAGEWOOD ESTATES","OFF VIGNES RD",BATON ROUGE,70805,30.488923,-91.158261

Task types

What this benchmark covers

Record-to-boundary joins1 task
Placing records in the correct county, district or watershed.
Facility and permit overlays2 tasks
Facilities, permits and enforcement actions on one map, from public records.
Area summaries1 task
Counts and totals by county or district, matching the underlying records.
Shareable output1 task
A map a colleague can open, filter and cite without rebuilding it.

Method

How we test

  1. 01

    Real agency deliverables

    One task per deliverable use case: trackers, dashboards, permit systems, forms, decks, workbooks, letters and one-pagers. Each carries its own rules and data, with problems planted where a careful reviewer would catch them.

  2. 02

    Every system can make files

    ChatGPT Astra 6.2, Claude Fable 5.2, Grok 4.7 and Gemini 4 Carbon run on their makers' own services with web search, high reasoning effort and a code tool whose files are what the user receives. When their code crashed or sent no files, they got up to two follow-ups. Justinian ran as shipped, with none.

  3. 03

    Graded as delivered

    Every file is rendered as the requester would open it and every app is opened in a browser. Code or instructions the user would still have to run, deploy or schedule do not count. Bill numbers and rule sections are checked against the official record, and every regulatory call is checked against the statute.

  4. 04

    Three blind judges

    Judges from OpenAI, Anthropic and Google, none of them a competing model, score each deliverable as the user would see it and rank all five systems with names masked. Every system is graded with the same checklist and scoring guide.

Briefing

Test Justinian on your agency's own work.

Request a briefing →

This page’s interactive features did not load. Reloading usually fixes it. You can email sales@vulcan.ai for demos or support@vulcan.ai for help.