Lab · Transparency
Runtime Release Tracking
Every score on this site was produced by an inference engine with an exact version. This page says which ones, tracks what has shipped upstream since, and commits us to never letting either change silently.
Transparency commitment · started 2026-08-22 · release data fetched automatically, last refreshed 2026-10-02
The one-paragraph version
A local-model benchmark has two version numbers that matter: the model’s and the inference engine’s. The second one is the one almost nobody tracks. Engines ship real performance and behavior changes — new kernels, new hardware support, sampler fixes — and runners can update the selected engine in the background with no prompt. That makes a silent engine update between rounds an invisible confound on every round-over-round comparison: a model can look faster, slower, or simply different when nothing about the model changed at all. So we pin the engines, record them per run, and publish both what we run and what has shipped since.
What the bench is running probed live 2026-10-02
Left column is what actually served our scores. Right column is what has shipped. A gap is not a bug — engines are pinned on purpose — but an unexplained gap is.
| Component | Running here | Latest | Why it matters here |
|---|---|---|---|
| LM Studio | 0.4.25+1 | 0.4.25 | Interactive / GUI runner. Served every score in the pre-2026-08-31 cohort. |
| Ollama | 0.35.0 | v0.35.0 | The unattended benchmarking runtime. Every nightly score is served by this. |
| MLX | [email protected] | v0.32.3 n/a | Apple's array framework. The MLX engine that serves every MLX-format run is built on it. |
| llama.cpp | [email protected] | b11344 n/a | The GGUF inference engine underneath both LM Studio's GGUF runtime and Ollama. |
Why one of those engine names says nax
LM Studio ships the MLX engine as two different packs under the same
version number: mlx-llm-mac-arm64-apple-metal-advsimd and
mlx-llm-mac-arm64-apple-metal-nax-advsimd. The
nax build is the one that uses the Neural Accelerators — the
matrix units Apple put inside every GPU core on M5. The plain build does not.
Every MLX score on this site was produced on the nax build, so which
one is selected is not cosmetic, it is part of what the number means. And because
both report 1.11.0, a swap between them is invisible to any check
that only compares version strings. We pin it explicitly and re-assert the pin on
every daemon start, after watching it revert twice in one day.
Every run archived by this bench records the runtime and engine versions that served it, so a score can always be traced back to the software that produced it. That is what makes a release note actionable: when an engine changes a sampler or a template, we can name the cells scored on the old side of the line rather than guessing.
n/a means the two columns use different version schemes and cannot be compared directly — LM Studio repackages llama.cpp and MLX under its own numbering, so pack 2.41.0 is not upstream build b11050. Older engine packs remain installed but unselected, so a regression can always be bisected against the engine that produced the original number.
The policy
- Runtime auto-update is OFF on the bench host. Engine versions do not change underneath a test series.
- Engine versions are captured per run. The runtime listing (exact package name and version, per model format) is recorded into the benchmark manifest, so every archived result names the engine that produced it.
- Engine updates get a deliberate review before adoption. When an update is pending, we read the release notes, assess what it could affect in prior results, adopt it at a round boundary — and log it in the changelog at the bottom of this page.
Releases
The last 10 releases for each product we depend on, pulled from the vendor’s own feed. Expand an entry for the summary; where upstream wrote more than fits here, the entry links to the official notes rather than paraphrasing them.
LM Studio
running 0.4.25+1
latest 0.4.25
Interactive / GUI runner. Served every score in the pre-2026-08-31 cohort.
0.4.25 2026-09-19
Build 1 - [Mac] Added support for the Splash engine on Apple Silicon M3 or newer running macOS 26.4 or newer.
0.4.24 2026-09-09
Build 1 - [LM Studio Engine Protocol] Added advanced llama.cpp argument overrides for GGUF model loading - Improve DSpark and DFlash assistant drafters compatibility heuristics - Fixed the Loaded Instances UI showing an incorrect context length - Fixed /api/v1/chat splitting text and image inputs into separate user messages - Improved sign-in reliability
0.4.23 2026-08-28
Build 1 - Improve reliability of Qwen 3.8 Flash Next
0.4.22 2026-08-28
Build 1 - Support for DFlash, DSpark, and MTP assistant drafters - Requires llama.cpp engine version 2.29.1 or newer - Added support for images returned by tools in OpenAI-compatible /v1/responses and /v1/chat/completions, and Anthropic-compatible /v1/messages - [Linux] Fixed AppImage startup on systems without FUSE 2 - [Linux] Fixed AppImage startup when Chromium sandboxing is unavailable - [Linux] Fixed lms wake-up for AppImage installations - [Linux] Fixed desktop icons for .deb installations - [Linux] Enabled in-app updates for AppImage and .deb installations - [Linux] Improved recovery after canceling .deb update authorization …
0.4.21+2 2026-08-12
Build 2 - Improved llama.cpp model-load error messages - Update advanced load settings for mmap, mlock, and direct io for llama.cpp engines 2.28.1 and newer. Build 1 - Support for local server API key in enterprise internal network endpoint mode
0.4.20 2026-07-22
Build 1 - Support for enterprise internal network model endpoint - Added support for using this device’s models in Bionic over LM Link - [LM Studio Engine Protocol] Added a developer setting to control the Llama.cpp engine log level
0.4.19+2 2026-07-07
Build 2 - Add /reasoning to lms chat - [STABLE] Default LM Studio Engine Protocol to ON. Build 1 - [LM Studio Engine Protocol] Fixed a bug where reasoning would be replayed as non-reasoning content while using /v1/responses or chat UI - [BETA] Default LM Studio Engine Protocol to ON.
0.4.18 2026-06-27
Build 1 - Fixed bug where zooming in and out could cause the interface to scale from the top-left and get clipped - Fixed a bug where reasoning on/off could be ignored by some models with custom reasoning settings - Fixed automatic chat title generation for some reasoning models - Added guidance in model settings for llama.cpp options that move when LM Studio Engine Protocol is enabled
0.4.17+4 2026-06-26
Build 4 - [LM Studio Engine Protocol] Fixed a bug where assistant output was replayed during a continuation request - [LM Studio Engine Protocol] Off by default, including for users upgrading from beta builds. You can re-enable it in Settings > Developer. - Add ability to open mermaid diagrams in full screen and save to PNG - Fix bug in /v1/responses with previous_response_id causing excessive prompt processing Build 3 - [LM Studio Engine Protocol] Existing custom Chat Template settings in My Models are now preserved when upgrading - [LM Studio Engine Protocol] Fixed a bug where unsupported GGUF prediction settings could appear in Chat, My Models, and Server configuration panels …
0.4.16+2 2026-06-08
Build 2 - Lm Link no longer requires waitlisting - Updated default context length to 8k tokens Build 1 - Introducing Locally, LM Studio's mobile app. Available on iPhone and iPad. - Use LM Link in Locally to take your largest LM Studio models on the go - Security hardening - [GGUF] Fix multi-GPU selection bugs affecting GPU ON/OFF and Priority Order on some CUDA 12, ROCm, and Vulkan setups
More releases → (earlier history lives on the vendor’s changelog; a browsable archive here is planned)
Ollama
running 0.35.0
latest v0.35.0
The unattended benchmarking runtime. Every nightly score is served by this.
v0.35.0 2026-09-28
Decision models
Ollama now supports decision models through /v1/systemone, based on TypeSafe’s Jev API.
Decision models return choices, probabilities, and scores instead of text. Use them for tasks such as ticket triage, model routing, and content classification.
Available models:
- Nimble from Bespoke Labs
- Tev1 from Together AI
sh
ollama pull nimble
Send context and one or more questions:
sh
curl http://localhost:11434/v1/systemone \
-H 'Content-Type: application/json' \
-d '{
"model": "nimble",
"state": "Our checkout has returned 500 errors since 9am.",
"questions": {
"label": {
"type": "choice" … v0.34.4 2026-09-23
What's Changed - Structured outputs on thinking models now apply in a single pass, making them faster and more reliable. - Fixed intermittent "model not found" errors with a large local library - Fixed the macOS app becoming unresponsive when checking if ChatGPT or Codex is running. - Qwen 3.8 prompt processing is faster on Apple Silicon. - Gemma 4 on Apple Silicon now picks the best image resolution per image, keeping more detail in high-resolution images. - Updated llama.cpp, MLX, and XGrammar.
v0.34.3 2026-09-19
What's Changed
GET /api/show now advertises each model's thinking controls and default:
Available in the CLI with:
ollama show gemma4
thinking
levels false, true
default true
Available in the API with:
sh
curl http://localhost:11434/api/show -d '{"model": "glm-5.3-flash:cloud"}'
json
{
"thinking": {
"values": ["low", "high", "max"],
"default": "max"
}
}
Also available on ollama.com directly for cloud models.
* Nemotron H vision models are now supported on Apple Silicon with MLX
* Ollama's macOS app will now no longer reopen windows you've closed when activating it
* Fix for model pulls from HuggingFace v0.34.2 2026-09-15
What's Changed - Added first-run setup when running ollama, with options to sign in or continue locally. Setup completion is shared with the desktop app on macOS and Windows. - Added ollama://apps to open the desktop app’s Apps page directly on macOS and Windows. - Fixed excessive memory growth during long generations with MLX speculative decoding. - Updated llama.cpp.
v0.34.1 2026-09-14
What's Changed * MLX safetensors ollama create no longer experimental. GGUF model creation now requires using llama.cpp tooling for safetensor conversion and quantization. * Improved MLX memory handling on Apple Silicon * Runaway repeat token detection now requires 100 repeat tokens for reduced false positives (e.g. OCR) * /api/tags is much faster on large model libraries (3.1 s → 294 ms cold in testing), and model capabilities are now reported consistently. * Deprecated typical_p: it can no longer be set when creating new models, existing GGUF models retain support. * MLX and llama.cpp updates
v0.34.0 2026-09-05
Use Ollama models in ChatGPT Desktop Ollama models can now be used directly in ChatGPT Desktop, so you can keep your existing workflow while running open models. Setup is available from the Ollama app on MacOS. <img width="1374" height="1300" alt="CleanShot 2026-09-08 at 11 04 07 AM@2x" src="https://github.com/user-attachments/assets/e10e299d-c11f-447d-9234-afa855824efe" /> This release also improves structured output performance on Apple Silicon, adds support for OpenAI-compatible client tool search and response compaction.
v0.33.3 2026-09-02
What's Changed * gemma4 now supports images and audio on MLX engine * Report cached prompt tokens * Honor GGUF model defined default parameters * MLX, MLX-C, llama.cpp update
v0.33.2 2026-08-27
What's Changed * Ollama's app now follows the system appearance again, restoring dark mode support * Fixed the macOS app to properly hand off to an already-running instance instead of starting a second one * The Claude Desktop proxy no longer interrupts in-flight requests when the model catalog updates
v0.33.1 2026-08-26
What's Changed * MLX: Qwen3.8 Flash Next support * cmake: make external compat patches idempotent * MLX and llama.cpp update * mlxrunner: add structured output support * mlxrunner: avoid Metal GPU timeouts when loading models from slow storage
v0.33.0 2026-08-21
What's Changed Claude Desktop Developers can now easily configure Claude Desktop to seamlessly work with Ollama as a third-party gateway provider. <img width="1298" height="780" alt="image" src="https://github.com/user-attachments/assets/c02fd93b-f599-40a9-83c3-d0555c0ac182" /> Improved caching * Fixed a hang where agent clients that cancel long prefills * Prefill restore points are now trustworthy by construction: a cancelled prefill keeps every restore point it crossed, so retries resume where they stopped instead of restarting from scratch …
More releases → (earlier history lives on the vendor’s changelog; a browsable archive here is planned)
MLX
running [email protected]
latest v0.32.3
Apple's array framework. The MLX engine that serves every MLX-format run is built on it.
v0.32.3 2026-09-29
What's Changed * Fix scan and sort for a zero-size axis by @eyupcanakman in https://github.com/ml-explore/mlx/pull/4340 * Add a correction parameter in std and var by @prady0t in https://github.com/ml-explore/mlx/pull/4348 * Fix tests/run.py stuck in macOS CI by @zcbenz in https://github.com/ml-explore/mlx/pull/4410 * Extension Example Updated by @jagrit06 in https://github.com/ml-explore/mlx/pull/4412 * Fix sorted gather_qmm NAX row overflow above 32K by @PhilipJohnBasile in https://github.com/ml-explore/mlx/pull/3922 * Fix deadlock caused by mx.clear_streams() holding GIL by @aleroot in https://github.com/ml-explore/mlx/pull/4413 …
v0.32.2 2026-08-25
What's Changed * Preserve subnormal float values when casting to bool by @reckylurker in https://github.com/ml-explore/mlx/pull/4224 * Support assigning through a bare Ellipsis index by @Adityaj0 in https://github.com/ml-explore/mlx/pull/4314 * Fix divmod truncating the quotient for floats by @ayaangazali in https://github.com/ml-explore/mlx/pull/4108 * Add force_fused option to scaled_dot_product_attention by @hojin12312 in https://github.com/ml-explore/mlx/pull/4185 * Reject a negative eps in the normalization layers by @ayaangazali in https://github.com/ml-explore/mlx/pull/4312 …
v0.32.1 2026-08-18
What's Changed * Fix int64 type cast error when loading GGUF metadata arrays by @danlee2002 in https://github.com/ml-explore/mlx/pull/3823 * Document default values in normalization layer docstrings by @Pablosinyores in https://github.com/ml-explore/mlx/pull/3819 * Warn at configure time when NAX kernels are disabled by @pierre427 in https://github.com/ml-explore/mlx/pull/3824 * [CUDA][Improvement] RMSNorm forward speed up by @nastya236 in https://github.com/ml-explore/mlx/pull/3850 * Fix captured random state in compile by @angeloskath in https://github.com/ml-explore/mlx/pull/3828 …
v0.32.0 2026-07-07
What's Changed * Generate qmm implementaions with cmake by @zcbenz in https://github.com/ml-explore/mlx/pull/3424 * Enable swap for all CI building CUDA by @zcbenz in https://github.com/ml-explore/mlx/pull/3437 * Bump minor by @angeloskath in https://github.com/ml-explore/mlx/pull/3438 * [CUDA] Fix qmm_naive K-tail dispatch for FP quantized kernels by @Lyxot in https://github.com/ml-explore/mlx/pull/3445 * Keep gguflib input-validation asserts active in release builds by @qflen in https://github.com/ml-explore/mlx/pull/3436 * Reuse nightly build's ccache for release by @zcbenz in https://github.com/ml-explore/mlx/pull/3458 …
v0.31.2 2026-04-22
Highlights * Wider support for cuda quantized matmuls (#3352, #3268, #3321, #3417, #3255) * MLX can be used by multiple threads for independent computations (#3405, #3348, #3281, #3423) * Added CUDA FFT support * JACCL is now a standalone lib (#3412) What's Changed * Bump by @angeloskath in https://github.com/ml-explore/mlx/pull/3244 * win: re-enable and fix cuDNN performance by @dhiltgen in https://github.com/ml-explore/mlx/pull/3242 * Fix crashes in multi-threaded process teardown by @louen in https://github.com/ml-explore/mlx/pull/3167 * [CUDA] Add FFT support by @lucasnewman in https://github.com/ml-explore/mlx/pull/3243 …
v0.31.1 2026-03-12
What's Changed * Bump the patch version by @angeloskath in https://github.com/ml-explore/mlx/pull/3185 * Skip Hopper-only kernels in CI by @zcbenz in https://github.com/ml-explore/mlx/pull/3184 * [CUDA] Fsdp (easy) by @nastya236 in https://github.com/ml-explore/mlx/pull/3130 * Fix ref leak in mx.save/load with file like object by @aisk in https://github.com/ml-explore/mlx/pull/3187 * Fix/missing libs in docs by @ChristophePRAT in https://github.com/ml-explore/mlx/pull/3190 * feat: adding the bartlett function by @Vlor999 in https://github.com/ml-explore/mlx/pull/3155 * Bump actions/download-artifact from 7 to 8 by @dependabot[bot] in https://github.com/ml-explore/mlx/pull/3189 …
v0.31.0 2026-02-28
Highlights * Initial version of QMMs for CUDA (#3160) * JACCL mesh bandwidth improvements (#3174) * Massive speedups for 3D convs (#3147) * Continued improvements to qqmm (#3106, #3022) What's Changed * Patch bump by @angeloskath in https://github.com/ml-explore/mlx/pull/3102 * is_available() should check the device index too by @andresy in https://github.com/ml-explore/mlx/pull/3107 * Fix residency set with user provided buffer by @awni in https://github.com/ml-explore/mlx/pull/3108 * Cleanup test_fast_sdpa.py by @zcbenz in https://github.com/ml-explore/mlx/pull/3112 * [CUDA] Set current device before allocating memory by @zcbenz in https://github.com/ml-explore/mlx/pull/3110 …
v0.30.6 2026-02-06
Highlights * Much faster bandwidth with JACCL on macOS >= 26.3 (some numbers) What's Changed * patch by @awni in https://github.com/ml-explore/mlx/pull/3093 * Disable managed memory on WSL when concurrentManagedAccess is not supported by @jessegross in https://github.com/ml-explore/mlx/pull/3095 * Fix non simd f16 build by @awni in https://github.com/ml-explore/mlx/pull/3097 * Fix 2pass sdpa on < M2 by @awni in https://github.com/ml-explore/mlx/pull/3099 * JACCL update by @angeloskath in https://github.com/ml-explore/mlx/pull/3094 * Fix qmv_impl for small N by @manuelcandales in https://github.com/ml-explore/mlx/pull/3096 …
v0.30.5 2026-02-03
What's Changed * patch by @awni in https://github.com/ml-explore/mlx/pull/3074 * [CUDA] Fallback Event impl when there is no hardware cpu/gpu coherency by @zcbenz in https://github.com/ml-explore/mlx/pull/3070 * Tune CUDA gaph sizes on B200 and H100 by @awni in https://github.com/ml-explore/mlx/pull/3077 * [Docs] Simple example of using MLX distributed by @stefpi in https://github.com/ml-explore/mlx/pull/2973 * Use lower-right causal mask alignment consistently by @Anri-Lombard in https://github.com/ml-explore/mlx/pull/2967 * Fix ALiBi slopes for non-power-of-2 num_heads by @vovw in https://github.com/ml-explore/mlx/pull/3071 …
v0.30.4 2026-01-27
Highlights - Metal: Much faster vector fused grouped-query attention for long context - CUDA: Several improvements to speed up LLM inference for CUDA backend - CUDA: Support for dense MoEs - CUDA: Better support for consumer GPUs (4090, 5090, RTX 6000, ...) What's Changed * patch bump for next release by @awni in https://github.com/ml-explore/mlx/pull/2991 * Fix fence by @awni in https://github.com/ml-explore/mlx/pull/2998 * Reverts changing the MLX_IBV_DEVICES to MLX_JACCL_DEVICES by @angeloskath in https://github.com/ml-explore/mlx/pull/2999 * fix distributed all_to_sharded bias shard axis from -2 to -1 by @gufengc in https://github.com/ml-explore/mlx/pull/2987 …
More releases → (earlier history lives on the vendor’s changelog; a browsable archive here is planned)
mlx-lm
latest v0.31.3
The MLX language-model layer — generation, quantization, and the conversion path most MLX weights on this bench came through.
v0.31.3 2026-04-22
Highlights * Lots of bugfixes * Thread local generation stream to accompany MLX v0.31.2 What's Changed * Bump the patch version by @angeloskath in https://github.com/ml-explore/mlx-lm/pull/1124 * Fix batch dimension mismatch in BatchKVCache and BatchRotatingKVCache extend() by @razorback16 in https://github.com/ml-explore/mlx-lm/pull/1141 * Fix parallel tool call handling in server by @kernelpool in https://github.com/ml-explore/mlx-lm/pull/1170 * Fix MiniMax M2 parallel tool calling by @kernelpool in https://github.com/ml-explore/mlx-lm/pull/1171 * Fix missing tree_reduce import in models/cache.py by @siiea-ai in https://github.com/ml-explore/mlx-lm/pull/1165 …
v0.31.2 2026-04-07
Highlights * Caching system prompt and user messages for non-trimmable caches * Batch generator refactoring What's Changed * Bump the patch version by @angeloskath in https://github.com/ml-explore/mlx-lm/pull/959 * Presence and frequency penalties by @angeloskath in https://github.com/ml-explore/mlx-lm/pull/971 * Eval self.left_padding whenever it is updated in BatchRotatingKVCache by @rltakashige in https://github.com/ml-explore/mlx-lm/pull/960 * Late binding caused incorrect cache checkpoint by @angeloskath in https://github.com/ml-explore/mlx-lm/pull/976 * Move to metal agnostic device_info by @angeloskath in https://github.com/ml-explore/mlx-lm/pull/979 …
v0.31.0 2026-03-07
What's Changed * Fix save/load of CacheList by @angeloskath in https://github.com/ml-explore/mlx-lm/pull/886 * Share model by @angeloskath in https://github.com/ml-explore/mlx-lm/pull/871 * Fix mixed quant predicates for MLA models by @spicyneuron in https://github.com/ml-explore/mlx-lm/pull/892 * Add JoyAI LLM Flash by @kernelpool in https://github.com/ml-explore/mlx-lm/pull/894 * perplexity: add --trust-remote-code option by @ivanfioravanti in https://github.com/ml-explore/mlx-lm/pull/896 * server: add usage.prompt_tokens_details.cached_tokens to json response by @percontation in https://github.com/ml-explore/mlx-lm/pull/849 …
v0.30.7 2026-02-12
What's Changed * Fix Kimi Linear by @kernelpool in https://github.com/ml-explore/mlx-lm/pull/853 * Bump version for next release by @awni in https://github.com/ml-explore/mlx-lm/pull/865 * Pythonic tool calling for LFM2 models by @viktike in https://github.com/ml-explore/mlx-lm/pull/864 * Fix DeepSeek V3.2 indexer and weight loading by @kernelpool in https://github.com/ml-explore/mlx-lm/pull/866 * Make validation set optional in training process by @Goekdeniz-Guelmez in https://github.com/ml-explore/mlx-lm/pull/857 * Mistral tool parser by @awni in https://github.com/ml-explore/mlx-lm/pull/874 * LongCat MLA by @kernelpool in https://github.com/ml-explore/mlx-lm/pull/868 …
v0.30.6 2026-02-04
What's Changed * Transformers v5 by @awni in https://github.com/ml-explore/mlx-lm/pull/811 * Add LongCat Flash tool parser by @kernelpool in https://github.com/ml-explore/mlx-lm/pull/810 * Add Kimi-K2.5 by @kernelpool in https://github.com/ml-explore/mlx-lm/pull/813 * Bump mlx version and version by @awni in https://github.com/ml-explore/mlx-lm/pull/816 * Fix NemotronH config compatibility with HuggingFace format by @LuqDaMan in https://github.com/ml-explore/mlx-lm/pull/820 * Fix for Exception - MultiLinear.to_quantized() missing 'mode' by @inferencers in https://github.com/ml-explore/mlx-lm/pull/809 …
v0.30.5 2026-01-25
What's Changed * import logging as it throws no logging error in place of actual error by @Maanas-Verma in https://github.com/ml-explore/mlx-lm/pull/778 * server: use OpenAI compatible finish_reason by @percontation in https://github.com/ml-explore/mlx-lm/pull/782 * move Xielu Activation in Apertus to activations.py by @Goekdeniz-Guelmez in https://github.com/ml-explore/mlx-lm/pull/772 * bump transformers by @awni in https://github.com/ml-explore/mlx-lm/pull/746 * Update glm4_moe_lite to store KV latent in cache by @N8python in https://github.com/ml-explore/mlx-lm/pull/780 * Adding TeleChat3 by @Goekdeniz-Guelmez in https://github.com/ml-explore/mlx-lm/pull/773 …
v0.30.4 2026-01-19
What's Changed * Add AWQ/GPTQ weight transformation utilities by @ericcurtin in https://github.com/ml-explore/mlx-lm/pull/730 * Add IQuest Coder V1 Loop variant by @kernelpool in https://github.com/ml-explore/mlx-lm/pull/716 * Fix sliding window batching by @awni in https://github.com/ml-explore/mlx-lm/pull/738 * Fix Batch Generation: Add extract method to ArraysCache for item retrieval by @Goekdeniz-Guelmez in https://github.com/ml-explore/mlx-lm/pull/740 * Make MambaCache compatible with batch generation for nemotron-h by @nikhilmitrax in https://github.com/ml-explore/mlx-lm/pull/690 * Add a server benchmark for continuous batching by @awni in https://github.com/ml-explore/mlx-lm/pull/728 …
v0.30.2 2026-01-06
What's Changed * Fix mlx-lm release by @awni in https://github.com/ml-explore/mlx-lm/pull/733
v0.30.1 2026-01-06
What's Changed * custom dsv32 chat template by @awni in https://github.com/ml-explore/mlx-lm/pull/693 * shard glm by @awni in https://github.com/ml-explore/mlx-lm/pull/698 * support minimax m2 by @awni in https://github.com/ml-explore/mlx-lm/pull/700 * Enhance load_config function to check for config file existence and i… by @cubist38 in https://github.com/ml-explore/mlx-lm/pull/701 * batch_generate fails with Phi3 (LongRoPE) when prompts have different lengths by @vyaivanove in https://github.com/ml-explore/mlx-lm/pull/707 * Fix GIL starvation in _generate thread when batch is idle by @sjug in https://github.com/ml-explore/mlx-lm/pull/706 …
v0.30.0 2025-12-18
What's Changed * fix: server busy-waiting during idle request polling by @zenyr in https://github.com/ml-explore/mlx-lm/pull/674 * Fixes for transformers v5 by @awni in https://github.com/ml-explore/mlx-lm/pull/684 * Add mimo v2 flash by @awni in https://github.com/ml-explore/mlx-lm/pull/685 * More useful error message for unsupported batching by @awni in https://github.com/ml-explore/mlx-lm/pull/687 * Model parallel generation by @angeloskath in https://github.com/ml-explore/mlx-lm/pull/676 * Bump to transformer v5 by @awni in https://github.com/ml-explore/mlx-lm/pull/689 * Revert return dict and wrap apply_chat_template by @awni in https://github.com/ml-explore/mlx-lm/pull/691 …
More releases → (earlier history lives on the vendor’s changelog; a browsable archive here is planned)
llama.cpp
running [email protected]
latest b11344
The GGUF inference engine underneath both LM Studio's GGUF runtime and Ollama.
llama.cpp does not cut semantic releases — it tags a build per merged commit, often ten or more a day. The ten below are therefore roughly the last day upstream, not the last year. What actually matters for our scores is the repackaged engine version in the table above.
b11344 pre-release 2026-10-02
<details open> CUDA: fix 2 broken Volta FA cases (#29803) </details> Website: - <https://llama.app> Attestations: - <https://github.com/ggml-org/llama.cpp/attestations/52093363> macOS/iOS: - macOS Apple Silicon (arm64) - macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED - macOS Intel (x64) - iOS XCFramework Linux: - Ubuntu x64 (CPU) - Ubuntu arm64 (CPU) - Ubuntu s390x (CPU) - Ubuntu x64 (Vulkan) - Ubuntu arm64 (Vulkan) - Ubuntu x64 (CUDA 12) - CUDA 12.8 libraries - Ubuntu x64 (CUDA 13) - CUDA 13.4 libraries - Ubuntu arm64 (CUDA 13) - CUDA 13.4 libraries - Ubuntu x64 (ROCm 10.0) - Ubuntu x64 (OpenVINO) - Ubuntu x64 (SYCL FP32) - Ubuntu x64 (SYCL FP16) …
b11342 pre-release 2026-10-02
<details open> common,rpc : fix cache dir creation through symlinks on buggy libstdc++ (#29816) See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101510 Close #29759 Signed-off-by: Adrien Gallouët <[email protected]> </details> Website: - <https://llama.app> Attestations: - <https://github.com/ggml-org/llama.cpp/attestations/52050509> macOS/iOS: - macOS Apple Silicon (arm64) - macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED - macOS Intel (x64) - iOS XCFramework Linux: - Ubuntu x64 (CPU) - Ubuntu arm64 (CPU) - Ubuntu s390x (CPU) - Ubuntu x64 (Vulkan) - Ubuntu arm64 (Vulkan) - Ubuntu x64 (CUDA 12) - CUDA 12.8 libraries - Ubuntu x64 (CUDA 13) - CUDA 13.4 libraries …
b11339 pre-release 2026-10-02
<details open> llama : clamp kpool re-pool bound to existing pools (#29805) * tests : simplify function signature * llama : clamp kpool re-pool bound to existing pools The n_tokens/kpool + n_seqs_unq bound on n_new_g overshoots when a batch fills the whole cache: n_ctx tokens complete exactly n_ctx/kpool pools, so the +1 pads new_pool_idxs/new_pool_rep one entry past n_pool_real. Graph reserve only covers n_pool_real entries, so the first full-context decode builds bigger tensors than reserved and ggml-alloc demands a graph reallocation (abort under GGML_SCHED_DEBUG_REALLOC=1). Clamp the bound to n_pool_real: a ubatch can never mark more pools than …
b11338 pre-release 2026-10-02
<details open> hexagon: shared strided DMA copy for CPY and CONCAT, any-dim CONCAT via DMA (#29685) * hexagon: shared strided DMA copy for CPY and CONCAT, any-dim CONCAT via DMA * hex-cpy: various fixes on top of the concat optimizations Removed CONCAT_DMA_MIN_ROW logic, it was broken with 64-bit DMA. While it's kinda silly to use DMA for tiny stuff if that tensor gets mapped to an extended buffer the only way to read it is DMA. Added missing dma_queue_flush() calls. Added additional guards for conditions we don't support. --------- Co-authored-by: Max Krasnyansky <[email protected]> </details> Website: - <https://llama.app> Attestations …
b11337 pre-release 2026-10-02
<details open> server: return HTTP 400 for invalid embedding requests (#29060) </details> Website: - <https://llama.app> Attestations: - <https://github.com/ggml-org/llama.cpp/attestations/52037093> macOS/iOS: - macOS Apple Silicon (arm64) - macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED - macOS Intel (x64) - iOS XCFramework Linux: - Ubuntu x64 (CPU) - Ubuntu arm64 (CPU) - Ubuntu s390x (CPU) - Ubuntu x64 (Vulkan) - Ubuntu arm64 (Vulkan) - Ubuntu x64 (CUDA 12) - CUDA 12.8 libraries - Ubuntu x64 (CUDA 13) - CUDA 13.4 libraries - Ubuntu arm64 (CUDA 13) - CUDA 13.4 libraries - Ubuntu x64 (ROCm 10.0) - Ubuntu x64 (OpenVINO) - Ubuntu x64 (SYCL FP32) - Ubuntu x64 (SYCL FP16) …
b11335 pre-release 2026-10-02
<details open> cuda : route sm70 to the Turing MMVQ nwarps table (#29753) * cuda : route sm70 to the Turing MMVQ nwarps table Volta (sm_70) has no MMVQ parameter table of its own and falls through to GENERIC, which launches K-quant batch-1 decode (ncols_dst == 1) at nwarps=4. sm_70 shares TURING's tuning: the K-quant vec_dot prefers nwarps=2 there. Route sm_70 to the existing MMVQ_PARAMETERS_TURING table in both the device and the host table selector. Measured on one Tesla V100 32GB PCIe (PG500-216, driver 580.178.04, CUDA 12.0.140) with Qwen3.8-27B Q4_K_M, tg128, interleaved A/B in 6 ABBA blocks with paired per-block deltas: +1.091 t/s = +3.17 % …
b11334 pre-release 2026-10-02
<details open> metal : release temporary private transfer buffers (#29777) * metal : release temporary private transfer buffers Assisted-by: OpenAI Codex * metal : fix order and formatting --------- Co-authored-by: Niklas Wenzel <[email protected]> </details> Website: - <https://llama.app> Attestations: - <https://github.com/ggml-org/llama.cpp/attestations/52026582> macOS/iOS: - macOS Apple Silicon (arm64) - macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED - macOS Intel (x64) - iOS XCFramework Linux: - Ubuntu x64 (CPU) - Ubuntu arm64 (CPU) - Ubuntu s390x (CPU) - Ubuntu x64 (Vulkan) - Ubuntu arm64 (Vulkan) - Ubuntu x64 (CUDA 12) - CUDA 12.8 libraries …
b11333 pre-release 2026-10-02
<details open> webgpu: add bfloat16 support for MUL_MAT/MUL_MAT_ID/GET_ROWS- #29358 (#29358) </details> Website: - <https://llama.app> Attestations: - <https://github.com/ggml-org/llama.cpp/attestations/52019658> macOS/iOS: - macOS Apple Silicon (arm64) - macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED - macOS Intel (x64) - iOS XCFramework Linux: - Ubuntu x64 (CPU) - Ubuntu arm64 (CPU) - Ubuntu s390x (CPU) - Ubuntu x64 (Vulkan) - Ubuntu arm64 (Vulkan) - Ubuntu x64 (CUDA 12) - CUDA 12.8 libraries - Ubuntu x64 (CUDA 13) - CUDA 13.4 libraries - Ubuntu arm64 (CUDA 13) - CUDA 13.4 libraries - Ubuntu x64 (ROCm 10.0) - Ubuntu x64 (OpenVINO) - Ubuntu x64 (SYCL FP32) …
b11332 pre-release 2026-10-02
<details open> llama : fix invalid assert in recurrent memory (#29799) </details> Website: - <https://llama.app> Attestations: - <https://github.com/ggml-org/llama.cpp/attestations/52014664> macOS/iOS: - macOS Apple Silicon (arm64) - macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED - macOS Intel (x64) - iOS XCFramework Linux: - Ubuntu x64 (CPU) - Ubuntu arm64 (CPU) - Ubuntu s390x (CPU) - Ubuntu x64 (Vulkan) - Ubuntu arm64 (Vulkan) - Ubuntu x64 (CUDA 12) - CUDA 12.8 libraries - Ubuntu x64 (CUDA 13) - CUDA 13.4 libraries - Ubuntu arm64 (CUDA 13) - CUDA 13.4 libraries - Ubuntu x64 (ROCm 10.0) - Ubuntu x64 (OpenVINO) - Ubuntu x64 (SYCL FP32) - Ubuntu x64 (SYCL FP16) …
b11331 pre-release 2026-10-02
<details open> CUDA: Handle compute type for NVFP4 on cublass path (#29173) * CUDA: Handle compute type for NVFP4 on cublass path Signed-off-by: ynankani <[email protected]> * Use BF16 compute type for quantized models if HW allows Signed-off-by: ynankani <[email protected]> * Set acc prec to bf16 for nvfp4 as it needs atleast bf16 range Signed-off-by: ynankani <[email protected]> * Update ggml/src/ggml-cuda/ggml-cuda.cu Co-authored-by: Johannes Gäßler <[email protected]> * preserve op_params for per-expert matmul Signed-off-by: ynankani <[email protected]> --------- Signed-off-by: ynankani <[email protected]> Co-authored-by: Johannes Gäßler <[email protected]> …
More releases → (earlier history lives on the vendor’s changelog; a browsable archive here is planned)
Unsloth / Unsloth Studio
latest v0.1.902-beta
Fine-tuning and local training stack. New to this bench — Studio is the desktop app around it. Not yet serving any score; tracked ahead of testing.
Not installed on the bench. Tracking started 2026-09-19, testing to follow.
v0.1.902-beta Command Palette + Desktop UI/UX 2026-10-01
This release brings faster navigation, shareable run settings, and clearer errors to Unsloth Desktop. It also 4x speeds up Laya decisions, expands hosted Decision API support, and keeps NVFP4, INT4, and MXFP4 checkpoints in 4-bit during LoRA training. Highlights - Command palette on Cmd/Ctrl+P. Also, you can now share your GGUF model's run settings - Laya decisions up to 4.1x faster, plus connections to TypeSafe and other hosted providers - Clearer errors that explain failed loads and generations, with a View logs button - NVFP4 LoRA with 45% lower peak memory on Qwen3.8-27B-NVFP4 - INT4 + MXFP4 checkpoints stay packed during LoRA, including Kimi-K2.7-Code and gpt-oss Desktop + chat …
v0.1.900-beta Laya Decision Models + Library 2026-09-28
We're adding support for decision models, a unified Library for docs and media, document viewer, many Apple Silicon improvements, creation of Skills, and ~4.5× faster image and video generation. * Run and serve Decision Models like Laya (open-source Jev) locally * Skills Editor to create, edit, and delete Skills directly in Desktop * ModelScope model downloading is now here for users who can't use HF * Library + Document Viewer for PDF, Word, Excel, PowerPoint, and chats. * You can now attach files before loading a model * Apple Silicon Improvements including batched serving, structured outputs, and TurboQuant KV cache. * Configure how much of a model each GPU receives. …
prebuilt-wheels-cu13 Flash-Attention2, Causal-Conv1D, Mamba_SSM Binaries 2026-09-27
Prebuilt Linux x86_64 CUDA 13 wheels for flash-attn 2.8.4, causal-conv1d 1.7.0 and mamba-ssm 2.3.2.post1, built for PyTorch 2.13 and 2.14 on Python 3.13.
v0.1.815-beta Qwen-Image-2.1 + Skills 2026-09-23
You can now run Qwen-Image-2.1 locally with Unsloth! This release also includes custom Agent Skills, and easier chat/project management. It also brings 2x faster reasoning blocks (60 FPS vs 30 FPS), more reliable training, and improved Linux installs and updates. Qwen Image 2.1 Guide Highlights * Fast FP8 Qwen-Image-2.1 + diffusers update + GGUF fixes * Qwen-Image-2.1 works for image editing and image gen! * Add custom Agent Skills to guide models through specific tasks. * Chats are now draggable/customizable in sidebar. * 2x faster long reasoning blocks (60 FPS vs 30 FPS before). Thinking UI/UX reworked. * In-app Debian updates and an Ubuntu 24.04+ installer for ARM64. …
v0.1.814-beta Qwen-Image-2.1 + Skills 2026-09-22
You can now run Qwen-Image-2.1 locally with Unsloth! This release also includes custom Agent Skills, and easier chat/project management. It also brings 2x faster reasoning blocks (60 FPS vs 30 FPS), more reliable training, and improved Linux installs and updates. Qwen Image 2.1 Guide Highlights * Qwen-Image-2.1 fixes - diffusers update + GGUF fix * Qwen-Image-2.1 works for image editing and image gen! * Add custom Agent Skills to guide models through specific tasks. * Chats are now draggable/customizable in sidebar. * 2x faster long reasoning blocks (60 FPS vs 30 FPS before). Thinking UI/UX reworked. * In-app Debian updates and an Ubuntu 24.04+ installer for ARM64. …
v0.1.813-beta Qwen-Image-2.1 + Skills 2026-09-22
You can now run Qwen-Image-2.1 locally with Unsloth! This release also includes custom Agent Skills, and easier chat/project management. It also brings 2x faster reasoning blocks (60 FPS vs 30 FPS), more reliable training, and improved Linux installs and updates. Qwen Image 2.1 Guide Highlights * Qwen-Image-2.1 support for image gen + more (Another update coming today for fixes!) * Add custom Agent Skills to guide models through specific tasks. * 2x faster long reasoning blocks (60 FPS vs 30 FPS before). Thinking UI/UX reworked. * Chats are now draggable/customizable in sidebar. * In-app Debian updates and an Ubuntu 24.04+ installer for ARM64. * Projects are easier to edit and organize. …
v0.1.812-beta Qwen-Image-2.1 + Skills 2026-09-22
We're releasing support for Qwen-Image-2.1, custom Agent Skills, and easier chat/project management. It also brings 2x faster reasoning blocks (60 FPS vs 30 FPS), more reliable training, and improved Linux installs and updates. Qwen Image 2.1 Guide Highlights * Qwen-Image-2.1 support for image generation and more * Add custom skills to guide models through specific tasks. * 2x faster long reasoning blocks (60 FPS vs 30 FPS before) * In-app Debian updates and an Ubuntu 24.04+ installer for ARM64. * Chats are now fully draggable in sidebar (web UI + Linux desktop app for now, full support coming very soon). Thinking UI/UX reworked. * Projects are easier to edit and organize. …
v0.1.811-beta Docker + Multi User + AMD Support 2026-09-18
We're releasing our new updated Docker image along with multi-user accounts, RDNA1+2, FP8/INT8 diffusion support, ARM64 CUDA Windows support and many training, GRPO and inference improvements. Also a Qwen3.8-Flash-Next 2x faster MTP hotfix. Highlights: * Qwen3.8-Flash-Next MTP hotfix (2x faster) from v0.1.810-beta * New Docker with NVIDIA & AMD support: Guide * Multi user accounts with isolation Settings > Accounts * INT8/FP8 Image Diffusion inference - 2x faster * ARM64 Windows CUDA support for training, inference * GRPO improvements - Qwen3.5 support + lastest TRL, vLLM support * RDNA1, RDNA2 support for AMD * Better Windows NVIDIA GPU detection and recovery Updated Docker image …
v0.1.810-beta Docker + Multi User + AMD Support 2026-09-17
We're releasing our new updated Docker image along with multi-user accounts, RDNA1+2, FP8/INT8 diffusion support, ARM64 CUDA Windows support and many training, GRPO and inference improvements Highlights: * New Docker with NVIDIA & AMD support: Guide * Multi user accounts with isolation Settings > Accounts * INT8/FP8 Image Diffusion inference - 2x faster * ARM64 Windows CUDA support for training, inference * GRPO improvements - Qwen3.5 support + lastest TRL, vLLM support * RDNA1, RDNA2 support for AMD * Better Windows NVIDIA GPU detection and recovery Updated Docker image …
Windows-ARM64 Windows ARM64 Binaries 2026-09-16
No release notes were published for this version.
More releases → (earlier history lives on the vendor’s changelog; a browsable archive here is planned) · Docs →
Engine changelog
One entry per engine event on this bench — what we adopted, when, and what it might mean for comparisons across the boundary. Upstream release history is above; this is our side of it.
- 2026-09-19 Caught the selected MLX engine silently reverting from the Neural-Accelerator build to the plain one, twice. Both packs report version 1.11.0, so nothing in any log looked wrong. Root cause was our own keep-alive script: it runs the LM Studio daemon every few minutes, and since app version 0.4.25 that daemon comes up selecting the plain pack. Re-pinning now happens in the same script, on the same cycle that was undoing it.
- 2026-09-19 Ollama upgraded on the bench: 0.32.15 to 0.34.2. Seven releases in that span, four of which touch scoring behavior rather than speed — structured-output support in the MLX runner (0.33.1), KV-cache and resumed-prefill fixes (0.33.0), a runaway-repeat-detection threshold raised to 100 tokens (0.34.1), and four separate llama.cpp updates underneath. Every Ollama score on this site to date was produced on 0.32.15 and sits on the old side of that boundary.
- 2026-09-19 Page rebuilt as a live release tracker. Release feeds for LM Studio, Ollama, MLX, mlx-lm, llama.cpp and Unsloth are now fetched automatically instead of hand-typed — the previous hand-maintained engine table had drifted twelve versions (llama.cpp pack 2.29.1 against an installed 2.41.0) in the 25 days since it was written.
- 2026-08-22 Began tracking; confirmed NA-aware MLX engine (1.11.0 nax) active for all MLX runs; llama.cpp Metal 2.29.1 for GGUF.
- next The next engine update will be reviewed and logged here before it serves a single scored run.