Lab · Transparency
Runtime Version Tracking
Every score on this site was produced by an inference engine with an exact version. This page says which ones, and commits us to never letting that change silently.
Transparency commitment · started 2026-08-22 · engine table and changelog updated with every change
Why engine versions matter
A local-model benchmark has two version numbers that matter: the model’s and the inference engine’s. The second one is the one almost nobody tracks. Engines ship real performance and behavior changes — new kernels, new hardware support, sampler fixes — and runners like LM Studio can auto-update the selected engine in the background, with no prompt and no visible notice.
That means a silent engine update between rounds is an invisible confound on every round-over-round comparison: a model can appear faster or slower, or even score differently, when nothing about the model changed at all. This is a real gap we found in our own setup — auto-update was on, a pending MLX engine bump (1.10.1 → 1.11.0) was sitting unannounced in the runtimes panel, and no run manifest we had ever produced recorded which engine build served it. We closed the gap, and this page is the public side of that fix.
The policy
- Runtime auto-update is OFF on the bench host. Engine versions do not change underneath a test series.
- Engine versions are captured per run. The runtime listing (exact package name and version, per model format) is recorded into the benchmark manifest, so every archived result names the engine that produced it.
- Engine updates get a deliberate review before adoption. When an update is pending, we read the release notes, assess what it could affect in prior results, adopt it at a round boundary — and log it in the changelog below.
Current engines captured live 2026-08-25
| Model format | Selected engine | Version | Notes |
|---|---|---|---|
| GGUF | llama.cpp-mac-arm64-apple-metal-advsimd | 2.29.1 | Metal llama.cpp engine — serves every GGUF-format run. |
| MLX | mlx-llm-mac-arm64-apple-metal-nax-advsimd | 1.11.0 | NA-aware MLX engine (note the nax segment) — serves every MLX-format run. |
Older engine versions remain installed but unselected (llama.cpp back to 2.24.0, MLX
back to 1.8.5, in both generic and nax variants) — useful if a
regression ever needs to be bisected against a prior engine.
Engine changelog
One entry per engine event — tracking milestones and every adopted update. Future engine bumps will appear here with what changed and what it might mean for comparisons across the boundary.
- 2026-08-22 Began tracking; confirmed NA-aware MLX engine (1.11.0 nax) active for all MLX runs; llama.cpp Metal 2.29.1 for GGUF.
- next The next engine update will be reviewed and logged here before it serves a single scored run.