Lab · Hardware
The M5 Neural Accelerator
Apple put a dedicated matrix engine in every M5 GPU core. The speedup is real — and lopsided. This is our running record of what it actually does to local-LLM inference numbers, and how it shapes the way we read our own.
Ongoing investigation · started 2026-08-22 · lab notebook — dated sections, updated as the picture changes
What it is 2026-08-22
With the M5 generation (M5, M5 Pro, M5 Max), Apple redesigned the GPU so that every core carries a dedicated Neural Accelerator — an on-die matrix engine for the dense matmuls transformer inference lives on, analogous to a discrete GPU’s tensor cores. Prior Apple GPUs ran matmul on general-purpose SIMD units; this is new, purpose-built silicon.
It’s exposed to developers through Metal 4’s Tensor Operations (TensorOps) and Metal Performance Primitives, and Apple updated MLX — their own array framework — to use these directly. One footnote matters operationally: MLX only engages the Neural Accelerators on macOS 26.2 or later. This is a framework-level capability, not any one app’s trick — anything built on a recent-enough MLX, running on 26.2+, should benefit.
The speedup is lopsided — and that’s the whole story 2026-08-22
Apple’s own benchmark (base M5 vs M4) shows the Neural Accelerator’s benefit is almost entirely a prefill/time-to-first-token win, not a decode-throughput win:
| Metric | Speedup (M5 vs M4) | Why |
|---|---|---|
| Time-to-first-token (TTFT) | 3.3× – 4.06× | Prefill is compute-bound — exactly what a matmul accelerator helps. |
| Generation speed (tok/s) | 1.19× – 1.27× | Decode is memory-bandwidth-bound, not compute-bound — a matmul engine can’t fix a bandwidth ceiling. The 19–27% gain tracks almost exactly with M5’s 28% memory-bandwidth increase over M4 (153 vs 120 GB/s), suggesting bandwidth, not the accelerator, is doing most of that work. |
That attribution isn’t just ours: an independent guide reached the same conclusion separately — decode gains on M5 “track almost linearly with the higher memory bandwidth.” Two independent sources landing on the same explanation is a good accuracy signal.
The implication for benchmarking: if you want to see the Neural Accelerator’s real effect, TTFT is the metric to watch, not tok/s. Tok/s-centric comparisons — which is most of what the local-LLM community publishes — are looking mostly at the metric the NA barely moves. Our own throughput axis is measured, and this finding is why we treat prefill and decode as separate stories rather than folding them into one speed number.
Confirmed: our MLX runs go through the NA-aware engine 2026-08-22
The bench host is an Apple M5 Max running LM Studio, on a macOS build above the 26.2 floor. LM Studio ships two separate MLX runtime packages: a generic MLX build, and an “Apple M5” build with explicit M5 Neural Accelerator support. We verified in the runtime manager that the M5-specific build is the one selected and active for the MLX slot — meaning every MLX-format run this bench has published was served through the NA-aware engine.
A second, independent confirmation comes from the engine’s own internal naming.
The runtime listing identifies the selected MLX engine as
mlx-llm-mac-arm64-apple-metal-nax[email protected] —
the nax segment (almost certainly “Neural Accelerator
eXtension” or similar) versus the plain
mlx-llm-mac-arm64-apple-metal-advsimd variant that isn’t selected.
The NA-aware build is active in the engine’s own package name, not just a GUI label.
There’s a structural reason this works without special effort on LM Studio’s
part: their open-source MLX engine is a thin wrapper directly around Apple’s own
mlx-lm package, built by the MLX team itself. NA support landed inside
MLX/mlx-lm, so the wrapper inherits it whenever the pinned dependency is
bumped — no from-scratch reimplementation involved.
Caveats we’re carrying 2026-08-22
- Apple’s numbers are base-M5. Their benchmark used a base M5 (24 GB) against M4. The relative TTFT-vs-decode pattern should hold architecturally on M5 Pro/Max, but the exact magnitudes on M5 Max — a much larger GPU with higher bandwidth — are untested by Apple’s post and unconfirmed by us.
- Our historical TTFT data can’t answer this retroactively. Apple’s methodology fixed the prompt at 4,096 tokens; our archived runs vary prompt length freely, so their TTFT samples swing with prompt size and can’t be read as an NA signal. A controlled fixed-prompt TTFT baseline is on the workbench.
- Metric weighting is an open question. If TTFT is where the hardware’s win actually shows up, runtime-and-hardware comparisons that lean on tok/s may under-credit (or miscredit) effects across the board. We’re deciding how much weight TTFT deserves in future runtime-comparison writeups.