Instrumentation & controls
Lab
The bench under the bench: instrumentation, hardware, and runtime findings that shape how we measure — published because a benchmark that doesn’t control its own confounds is just vibes.
Scores are only as trustworthy as the machinery that produced them. This section is the running record of what we’ve learned about that machinery — the silicon, the inference engines, the metrics themselves — and the controls we’ve put in place when a finding exposed a gap. Investigations stay open; entries get dated updates as the picture changes.
PocketPal AI on iPhone
The open-source llama.cpp app that treats an iPhone like a bench. Every setting explained, which four decide whether a run is comparable, how to read its memory warning, and what we are measuring on an iPhone 18 Pro Max and an iPad Pro M5 against our Mac baselines.
HardwareThe M5 Neural Accelerator
Apple put a matrix engine in every GPU core. The speedup is real — and lopsided. Where it actually shows up in inference numbers, why decode barely moves, and how we confirmed our own MLX runs are served by the NA-aware engine.
TransparencyRuntime Release Tracking
A silent engine update between rounds is an invisible confound on every round-over-round comparison. Live release feeds for LM Studio, Ollama, MLX, mlx-lm, llama.cpp and Unsloth, what the bench is actually running against each, and the policy that keeps the two from diverging quietly.
More investigations will land here as they mature: memory-pressure telemetry during long runs, prompt-length-controlled TTFT baselines, and effort-setting sensitivity in reasoning models are all on the workbench.