Ornith-1.0-35B 4bit MLX

Retired Rank #22 of 53 · 7/8 GAUNTLET progress

Hollow markers & dashed spokes: axis not yet scored

86.3

94.7

90.2

n/d

90.4

35.8

Specification

Parameters: 35B total (MoE; active-per-token not confirmed in model card)
Architecture: qwen3_5_moe
Size on disk: 20.4 GB
Quantization: 4bit
Format: MLX
Runtime: LM Studio
Class: Standard
Reasoning (CoT): Yes — emits reasoning tokens
Internal ID: M28
Mean speed: 122.9 tok/s across suites
Model card: huggingface.co

Suite results

Suite Score Avg / 20 Tests n tok/s Run

General capability (13-task real-workload suite) 1a 224.4 / 260 18.7 12/13 ● n=1.1 124.4 run 2026-08-22

Test	Category	Score	tok/s
A1	Dev/Ops Scripting	14/20	117.5
A2	Dev/Ops Scripting	14/20	124
A3	Dev/Ops Scripting	20/20	121.6
A4	Dev/Ops Scripting	20/20	121.4
A5	Dev/Ops Scripting	20/20	128.3
A6	Dev/Ops Scripting	20/20	121.1
B1	Document Processing	19/20	123.8
B2	Document Processing	20/20	121.5
C1	Content Production	20/20	112.2
C2	Content Production	18/20	109.7
C3	Content Production	19/20	111.5
C4	Content Production	20/20	113.7

Category mean Dev/Ops Scripting 18Document Processing 19.5Content Production 19.3

Agentic tool-calling & protocol adherence 1b 156.8 / 160 19.6 8/8 n=1 134.6 run 2026-08-22

Test	Category	Score	tok/s
WA1	Tool Calling & Protocol Adherence	20/20	109.3
WA2	Tool Calling & Protocol Adherence	20/20	99.3
WA3	Tool Calling & Protocol Adherence	20/20	114.1
WA4	Tool Calling & Protocol Adherence	20/20	83
WA5	Tool Calling & Protocol Adherence	20/20	115.3
WB1	Agent Robustness & State Management	20/20	127.3
WB2	Agent Robustness & State Management	20/20	126.8
WB3	Agent Robustness & State Management	17/20	123.4

Category mean Tool Calling & Protocol Adherence 20Agent Robustness & State Management 19

Coding depth 1c 178.2 / 180 19.8 9/9 n=1 128.9 run 2026-09-19

Test	Category	Score	tok/s
K1	Coding — Concurrency	20/20	129.5
K2	Coding — Algorithms	20/20	133.2
K3	Coding — Security Review	18/20	131.4
K4	Coding — Performance	20/20	131.8
K5	Coding — Refactoring Under Constraints	20/20	130.7
K6	Coding — Test Writing	20/20	123.2
K7	Coding — React Component (typed)	20/20	118.9
K8	Coding — TypeScript Type System	20/20	124.3
K9	Coding — React Debugging	20/20	124.2

Category mean Coding — Concurrency 20Coding — Algorithms 20Coding — Security Review 18Coding — Performance 20Coding — Refactoring Under Constraints 20Coding — Test Writing 20Coding — React Component (typed) 20Coding — TypeScript Type System 20Coding — React Debugging 20

Doc/OCR vision 1d 160 / 160 20 8/8 n=1 125.7 run 2026-09-18

Test	Category	Score	tok/s
D1	Doc/OCR — Transcription	20/20	71.9
D2	Doc/OCR — Document QA	20/20	80.6
D3	Doc/OCR — Field Extraction	20/20	85.2
D4	Doc/OCR — Document QA	20/20	106.7
D5	Doc/OCR — Transcription	20/20	106
D6	Doc/OCR — Field Extraction	20/20	98.3
D7	Doc/OCR — Degraded Input	20/20	90.7
D8	Doc/OCR — Degraded Input	20/20	110.8

Category mean Doc/OCR — Transcription 20Doc/OCR — Document QA 20Doc/OCR — Field Extraction 20Doc/OCR — Degraded Input 20

Doc/OCR — real-degraded tier 1d2 67.2 / 80 16.8 4/4 n=1 121.4 run 2026-09-17

Test	Category	Score	tok/s
D9	Doc/OCR — Real Degraded Transcription	15/20	84.6
D10	Doc/OCR — Handwriting Extraction	18/20	83.7
D11	Doc/OCR — Degraded Table Transcription	14/20	112
D12	Doc/OCR — Degraded Table QA	20/20	103.6

Category mean Doc/OCR — Real Degraded Transcription 15Doc/OCR — Handwriting Extraction 18Doc/OCR — Degraded Table Transcription 14Doc/OCR — Degraded Table QA 20

Content-production depth 1e 115.8 / 120 19.3 6/6 n=1 123.1 run 2026-09-18

Test	Category	Score	tok/s
E1	SOW & Requirements Decomposition	20/20	129.5
E2	SOW & Requirements Decomposition	20/20	128
E3	Executive & Proposal Content	18/20	119.4
E4	Executive & Proposal Content	20/20	114
E5	Executive & Proposal Content	20/20	118.1
E6	Executive & Proposal Content	18/20	121.6

Category mean SOW & Requirements Decomposition 20Executive & Proposal Content 19

Long-context retrieval & synthesis 1g 118.2 / 120 19.7 6/6 n=1 116.3 run 2026-09-18

Test	Category	Score	tok/s
L1	Long-Context — Needle Retrieval	20/20	93.3
L2	Long-Context — Grounded QA with Distractors	20/20	76.4
L3	Long-Context — Ops Log Reasoning	18/20	68.9
L4	Long-Context — Faithful Summarization	20/20	94.7
L5	Long-Context — Instruction Retention	20/20	72.9
L6	Long-Context — Full-Haystack QA	20/20	39

Category mean Long-Context — Needle Retrieval 20Long-Context — Grounded QA with Distractors 20Long-Context — Ops Log Reasoning 18Long-Context — Faithful Summarization 20Long-Context — Instruction Retention 20Long-Context — Full-Haystack QA 20

Long-context multi-needle (MRCR) 1g2 44.1 / 60 14.7 3/3 n=1 101.7 run 2026-09-16

Test	Category	Score	tok/s
L7	Long-Context — Multi-Needle Retrieval (MRCR)	20/20	93.2
L8	Long-Context — Multi-Needle Retrieval (MRCR)	20/20	61.2
L9	Long-Context — Multi-Needle Retrieval (MRCR)	4/20	48.8

Live one-shot builds (runtime-verified) 1h 72.8 / 80 18.2 4/4 n=1 123.3 run 2026-09-16

Test	Category	Score	tok/s
H1	Web-Estate One-Shot Builds	19/20	126.2
H2	Web-Estate One-Shot Builds	19/20	127
H3	Web-Estate One-Shot Builds	17/20	122
H4	Web-Estate One-Shot Builds	18/20	114.3

Brand-constrained static build (negative-constraint adherence) 1j 90 / 100 18 5/5 n=1 129.9 run 2026-09-15

Test	Category	Score	tok/s
J1	Brand-Constrained Static Build	20/20	129.3
J3	Brand-Constrained Static Build	20/20	131.4
J4	Brand-Constrained Static Build	20/20	124.4
J5	Brand-Constrained Static Build	12/20	124.3
J6	Brand-Constrained Static Build	18/20	123.8

Each test is LLM-judged 0–20 against a fixed rubric; suite max = tests × 20. An ADJ chip marks a cell carrying a subjective adjustment from the maintainer; the judged score is the one shown, is unchanged, and remains what the axes, ranks and tie bands are computed from. Hover for the reason. Rows with a ▶ expand to the per-test breakdown. Where fewer tests ran than the suite total, unrun tests count as zero toward the axis score. See methodology.