Suite results
Suite Score Avg / 20 Tests n tok/s Run
▶ General capability (13-task real-workload suite) 1a 227.5 / 260 17.5 13/13 n=1.5 21.1 run 2026-10-04
| Test | Category | Score | | tok/s |
| A1 | Dev/Ops Scripting | 0/20 | | 20.5 |
| A2 | Dev/Ops Scripting | 19/20 | | 22.5 |
| A3 | Dev/Ops Scripting | 20/20 | | 21.8 |
| A4 | Dev/Ops Scripting | 11/20 | | 21.5 |
| A5 | Dev/Ops Scripting | 20/20 | | 21.3 |
| A6 | Dev/Ops Scripting | 20/20 | | 21.6 |
| B1 | Document Processing | 20/20 | | 21.9 |
| B2 | Document Processing | 20/20 | | 21.1 |
| B3 | Document Processing | 17/20 | | 19.9 |
| C1 | Content Production | 20/20 | | 20.7 |
| C2 | Content Production | 20/20 | | 20.5 |
| C3 | Content Production | 20/20 | | 19.8 |
| C4 | Content Production | 20/20 | | 19.8 |
Category mean Dev/Ops Scripting 15Document Processing 19Content Production 20
▶ Agentic tool-calling & protocol adherence 1b 160 / 160 20 8/8 n=1 28.5 run 2026-10-04
| Test | Category | Score | | tok/s |
| WA1 | Tool Calling & Protocol Adherence | 20/20 | | 28.4 |
| WA2 | Tool Calling & Protocol Adherence | 20/20 | | 29 |
| WA3 | Tool Calling & Protocol Adherence | 20/20 | | 28.6 |
| WA4 | Tool Calling & Protocol Adherence | 20/20 | | 24.7 |
| WA5 | Tool Calling & Protocol Adherence | 20/20 | | 28.5 |
| WB1 | Agent Robustness & State Management | 20/20 | | 21.8 |
| WB2 | Agent Robustness & State Management | 20/20 | | 18.8 |
| WB3 | Agent Robustness & State Management | 20/20 | | 17.9 |
Category mean Tool Calling & Protocol Adherence 20Agent Robustness & State Management 20
▶ Coding depth 1c 60.3 / 180 6.7 9/9 n=1 20.8 run 2026-10-07
| Test | Category | Score | | tok/s |
| K1 | Coding — Concurrency | 0/20 | | 20.9 |
| K2 | Coding — Algorithms | 20/20 | | 21.7 |
| K3 | Coding — Security Review | 0/20 | | 20.4 |
| K4 | Coding — Performance | 0/20 | | 20.5 |
| K5 | Coding — Refactoring Under Constraints | 20/20 | | 20.7 |
| K6 | Coding — Test Writing | 20/20 | | 21 |
| K7 | Coding — React Component (typed) | 0/20 | | 20.4 |
| K8 | Coding — TypeScript Type System | 0/20 | | 20.8 |
| K9 | Coding — React Debugging | 0/20 | | 20.4 |
Category mean Coding — Concurrency 0Coding — Algorithms 20Coding — Security Review 0Coding — Performance 0Coding — Refactoring Under Constraints 20Coding — Test Writing 20Coding — React Component (typed) 0Coding — TypeScript Type System 0Coding — React Debugging 0
▶ Doc/OCR vision 1d 160 / 160 20 8/8 n=1 23.1 run 2026-10-05
| Test | Category | Score | | tok/s |
| D1 | Doc/OCR — Transcription | 20/20 | | 16.2 |
| D2 | Doc/OCR — Document QA | 20/20 | | 13.6 |
| D3 | Doc/OCR — Field Extraction | 20/20 | | 17 |
| D4 | Doc/OCR — Document QA | 20/20 | | 18.1 |
| D5 | Doc/OCR — Transcription | 20/20 | | 14.8 |
| D6 | Doc/OCR — Field Extraction | 20/20 | | 14.1 |
| D7 | Doc/OCR — Degraded Input | 20/20 | | 17.9 |
| D8 | Doc/OCR — Degraded Input | 20/20 | | 19.1 |
Category mean Doc/OCR — Transcription 20Doc/OCR — Document QA 20Doc/OCR — Field Extraction 20Doc/OCR — Degraded Input 20
▶ Doc/OCR — real-degraded tier 1d2 59.2 / 80 14.8 4/4 n=1 19.3 run 2026-10-05
| Test | Category | Score | | tok/s |
| D9 | Doc/OCR — Real Degraded Transcription | 14/20 | | 19 |
| D10 | Doc/OCR — Handwriting Extraction | 17/20 | | 18 |
| D11 | Doc/OCR — Degraded Table Transcription | 13/20 | | 19 |
| D12 | Doc/OCR — Degraded Table QA | 15/20 | | 19 |
Category mean Doc/OCR — Real Degraded Transcription 14Doc/OCR — Handwriting Extraction 17Doc/OCR — Degraded Table Transcription 13Doc/OCR — Degraded Table QA 15
▶ Content-production depth 1e 120 / 120 20 6/6 n=1 21.2 run 2026-10-07
| Test | Category | Score | | tok/s |
| E1 | SOW & Requirements Decomposition | 20/20 | | 21.7 |
| E2 | SOW & Requirements Decomposition | 20/20 | | 21.7 |
| E3 | Executive & Proposal Content | 20/20 | | 20.8 |
| E4 | Executive & Proposal Content | 20/20 | | 21.1 |
| E5 | Executive & Proposal Content | 20/20 | | 20.4 |
| E6 | Executive & Proposal Content | 20/20 | | 21 |
Category mean SOW & Requirements Decomposition 20Executive & Proposal Content 20
▶ Long-context retrieval & synthesis 1g 120 / 120 20 6/6 n=1 20.2 run 2026-10-05
| Test | Category | Score | | tok/s |
| L1 | Long-Context — Needle Retrieval | 20/20 | | 13.9 |
| L2 | Long-Context — Grounded QA with Distractors | 20/20 | | 14.6 |
| L3 | Long-Context — Ops Log Reasoning | 20/20 | | 10.7 |
| L4 | Long-Context — Faithful Summarization | 20/20 | | 14 |
| L5 | Long-Context — Instruction Retention | 20/20 | | 8.2 |
| L6 | Long-Context — Full-Haystack QA | 20/20 | | 2.4 |
Category mean Long-Context — Needle Retrieval 20Long-Context — Grounded QA with Distractors 20Long-Context — Ops Log Reasoning 20Long-Context — Faithful Summarization 20Long-Context — Instruction Retention 20Long-Context — Full-Haystack QA 20
▶ Long-context multi-needle (MRCR) 1g2 44.1 / 60 14.7 3/3 n=1 16.3 run 2026-10-05
| Test | Category | Score | | tok/s |
| L7 | Long-Context — Multi-Needle Retrieval (MRCR) | 20/20 | | 11.6 |
| L8 | Long-Context — Multi-Needle Retrieval (MRCR) | 20/20 | | 6.1 |
| L9 | Long-Context — Multi-Needle Retrieval (MRCR) | 4/20 | | 8.8 |
▶ Live one-shot builds (runtime-verified) 1h 20 / 80 5 4/4 n=1 18 run 2026-10-05
| Test | Category | Score | | tok/s |
| H1 | Web-Estate One-Shot Builds | 0/20 | | 18.2 |
| H2 | Web-Estate One-Shot Builds | 0/20 | | 18 |
| H3 | Web-Estate One-Shot Builds | 0/20 | | 17.9 |
| H4 | Web-Estate One-Shot Builds | 20/20 | | 18 |
▶ Production replay (real agent workload) 1i 57.6 / 240 4.8 12/12 n=1 17.5 run 2026-10-10
| Test | Category | Score | | tok/s |
| I1 | Nightly Fix Replay | 0/20 | | 17.9 |
| I2 | Nightly Fix Replay | 0/20 | | 16.1 |
| I3 | Nightly Fix Replay | 8/20 | | 12.5 |
| I4 | Nightly Fix Replay | 0/20 | | 15.2 |
| I5 | Nightly Fix Replay | 0/20 | | 10.7 |
| I6 | Nightly Fix Replay | 0/20 | | 18.1 |
| I7 | Audit Judgment | 10/20 | | 19.4 |
| I8 | Audit Judgment | 0/20 | | 19.5 |
| I9 | Cron Dependency Reasoning | 20/20 | | 18.8 |
| I10 | Cron Dependency Reasoning | 0/20 | | 18.2 |
| I11 | Curation Judgment | 20/20 | | 19 |
| I12 | Curation Judgment | 0/20 | | 19.1 |
Category mean Nightly Fix Replay 1.3Audit Judgment 5Cron Dependency Reasoning 10Curation Judgment 10
▶ Brand-constrained static build (negative-constraint adherence) 1j 30 / 100 6 5/5 n=1 21.4 run 2026-10-10
| Test | Category | Score | | tok/s |
| J1 | Brand-Constrained Static Build | 10/20 | | 21.9 |
| J3 | Brand-Constrained Static Build | 0/20 | | 21 |
| J4 | Brand-Constrained Static Build | 20/20 | | 22 |
| J5 | Brand-Constrained Static Build | 0/20 | | 20.8 |
| J6 | Brand-Constrained Static Build | 0/20 | | 20.5 |
Each test is LLM-judged 0–20 against a fixed rubric; suite max = tests × 20.
An ADJ chip marks a cell carrying a subjective
adjustment from the maintainer; the judged score is the one shown, is unchanged,
and remains what the axes, ranks and tie bands are computed from. Hover for the reason.
Rows with a ▶ expand to the per-test breakdown.
Where fewer tests ran than the suite total, unrun tests count as zero toward the axis score.
See methodology.