llama.cpp Boosts Multimodal and MTP Performance

KV Cache Is Becoming the Memory Hierarchy of Inference

A recent pull request to the ggml-org/llama.cpp repository (PR #23269) introduces MTP-related improvements for the LLaMA C++ inference engine. The update, shared by a Reddit user, targets performance and/or memory enhancements tied to MTP (mixed-precision / tensor processing) that could speed local LLaMA model inference and efficiency. This matters because llama.cpp is a core open-source runtime used to run LLaMA-family models on local hardware, and efficiency gains directly affect latency, hardware requirements, and energy use for developers and hobbyists. Users running local LLaMA instances should review and merge the changes to benefit from improved throughput and resource use.

Mac users are debating whether to stick with MLX quantized models or switch to GGUF with MTP, as benchmarks for token generation and prompt processing vary. The post notes LM Studio handles MLX poorly due to bad caching and lack of MTP, while omlx offers strong caching, turboquant and dflash but also lacks MTP (which may arrive soon). GGUF gains MTP support that can accelerate multi-threaded processing, potentially improving throughput on macOS, while tooling and cache behaviors across runtimes (LM Studio, omlx) influence real-world performance. The choice matters for mac users optimizing local LLM inference latency and efficiency depending on model format, runtime features, and upcoming MTP support.

The ggml-org/llama.cpp project released version b9200, introducing changes to prompt processing that avoid copying logits for every token in a batch during multi-token processing (MTP). The update — linked on the project's GitHub release page and discussed in a recent comment by user am17an — aims to improve efficiency and could increase tokens-per-second or prompt-processing performance (pp) for batched inference. This matters to developers and researchers running LLaMA-family models locally or on resource-constrained hardware because reduced memory operations can lower latency and CPU/GPU overhead. The release signals incremental optimization in popular open-source ML runtimes, benefiting deployment and experimentation.

A hands-on test shows running large models locally may not be cheaper than using cloud inference. Williamangel benchmarked Gemma 4 31B on a 14" M5 Max MacBook (64GB, $4,299) and observed 10–40 token/s local throughput; accounting for electricity ($0.18–$0.20/kWh) yields $0.40–$4.79 per million tokens. By contrast, OpenRouter cloud instances serving the same model cost $0.38–$0.50 per million tokens and some providers deliver 60–70 token/s—faster and often cheaper. The analysis argues proponents of local inferencing ignore hardware purchase, depreciation, idle time, and other operational costs; perceived benefits like one‑time payment and privacy drive the myth that local equals cheaper. The piece reframes cost comparisons for AI deployments.

A user reports switching from Qwen 3.6 35B A3B to Qwen 3.6 27B to try the new MTP speculative draft and shares their llama-server launch flags. They list model path, context length, number of layers to keep low-level (ngl), parallelism, thread count, jinja, host/port, reasoning budget, and speculative type set to draft-mtp, but the snippet cuts off before spec-draft-n-max. The post seeks experiences or tuning tips for MTP speculative sampling on a 27B Qwen model—important for practitioners optimizing latency and throughput when running medium-large LLMs locally or on servers. This matters to developers and sysadmins deploying inference pipelines and exploring speculative decoding trade-offs.

A hands-on cost analysis finds running LLMs on an M5 Max MacBook Pro is generally more expensive than using cloud-hosted OpenRouter models. The author compares electricity (~$0.18–$0.20/kWh), hardware amortization for a $4,299 M5 Max laptop over 3–10 years, and measured token throughput (10–40 tokens/sec) to compute per-million-token costs. Depending on lifespan, power draw and throughput, on-device inference ranges roughly $0.40–$4.79 per million tokens, while OpenRouter’s Gemma 4 31b runs about $0.38–$0.50 per million tokens and is often 2x faster. Conclusion: hardware amortization dominates local costs and cloud inference is typically cheaper and faster, though running advanced models locally on consumer silicon is increasingly viable.

Benchmarks posted on Reddit show Qwen 3.6-27B Dense running with MTP (Mixture of Textual Prompts?) on an ASUS Strix Halo Windows setup, reporting performance and inference behavior. The post links to an image with benchmark results, implying plug-and-play testing of the Qwen 27B dense model under Windows using local LLM tooling. This matters to developers and hobbyists deploying large open models locally because it provides real-world performance reference for a recent Qwen variant on consumer-grade hardware and may inform choices about model size, runtime settings, and hardware compatibility. The content is practical for people evaluating local inference latency, memory usage, and setup details for running Qwen models on Windows.

Benchmarks show Strix Halo’s multi-token prefetch (MTP) implementation significantly speeds up Llama.cpp inference for some models: a 27B-MTP build doubled generation throughput (7.63 → 16.15 tokens/sec) and cut end-to-end wall time by ~11% versus the standard 27B in single-turn 15k-token tests, though prompt throughput fell ~12%. Results for the 35B model were mixed, indicating MTP’s benefits depend on model size and runtime trade-offs. Tests used Qwen 3.6 weights across builds and measured prompt t/s, generation t/s, and wall time. This matters for edge and local inference and developers tuning Llama.cpp or Strix Halo for latency/throughput trade-offs in on-device or self-hosted LLM deployments.

Support for MTP (Masked Transformer Pruning) has been merged into the main branch of the llama.cpp repository via PR #22673. The change, merged by contributors to the ggml-org/llama.cpp project, integrates MTP into the popular open-source C/C++ implementation used for running LLaMA-family models locally. This matters because llama.cpp is widely used to run efficient, on-device LLMs; MTP can improve inference efficiency and model size trade-offs, making it easier for developers and hobbyists to run pruned or optimized transformer variants without heavy infrastructure. The merge signals continued community-driven enhancements to local LLM tooling and potential performance gains for lightweight deployments.

youssofal/MTPLX: 2.24x decode TPS increase On Qwen 3.6 27B @ temp 0.6 | Native MTP Speculative Decoding On Apple Silicon With No External

A developer reports that running Gemma 4 with LiteRT-LM on mobile significantly outperforms their previous llama.cpp setup in memory usage and speed. They tested edge AI on-device inference, finding Gemma 4 plus LiteRT-LM reduced RAM footprint and improved latency compared with older Gemma 3 and llama.cpp builds, making practical local AI more feasible for everyday tasks. The post highlights implications for on-device AI adoption: lower resource models and runtime optimizations can enable richer local applications, better privacy, and reduced cloud dependence. This matters to mobile AI developers, runtime maintainers, and startups aiming for efficient offline inference.

A Reddit post demonstrates an automated AI researcher running entirely locally using llama.cpp and local models, showcasing autonomous task orchestration without cloud APIs. The demo chains prompts, tool use and memory to perform research-like workflows on a user’s machine, highlighting privacy, cost and latency advantages over cloud-hosted agents. It matters because lightweight C/C++ runtimes like llama.cpp enable complex agent behavior on commodity hardware, expanding access to autonomous AI workflows for developers and hobbyists while raising questions about safety, model provenance and resource limits. The post signals growing maturity of local LLM tooling and could accelerate experiments in offline agents, self-driving research assistants and privacy-preserving AI development.

Developers have implemented Multi-Token Prediction (MTP) for Qwen models running on LLaMA.cpp with TurboQuant, enabling the model to predict multiple tokens per forward pass on CPU-bound, quantized setups. The mod integrates MTP into the LLaMA.cpp runtime, adapts TurboQuant quantization formats, and demonstrates throughput and latency gains on local deployments, notably benefiting users running large Qwen variants without GPUs. This matters because it improves efficiency and responsiveness of local LLM inference, lowering compute cost and widening access for developers, hobbyists, and edge deployments. The post includes implementation details, benchmarks, and compatibility notes for quantization formats and prompts, guiding adopters on trade-offs and setup steps.

A user reports running large Mixture-of-Experts models—Qwen 3.6 35B-A3B and Gemma 4 26B-A4B—on a $200 secondhand PC (i7-6700, GTX 1080, 32 GB RAM) using llama.cpp with TurboQuant/RotorQuant KV-cache quantization to fit a 128k context in 8 GB VRAM. They claim throughput exceeding 24 tokens/sec and show benchmark tables for Q4_K_M quantized builds, demonstrating multi-expert inference on constrained hardware by offloading and compressing KV cache. This matters because it lowers the hardware barrier for running large-context MoE and dense LLMs, enabling researchers and hobbyists to experiment with massive models and long-context workloads without high-end GPUs or cloud costs. The post highlights practical quantization and engineering tricks rather than new model releases.

Developer released Docker images for llama.cpp that include recent MTP (Multi-Token Programming) PR improvements—notably image support and bug fixes—so users can run MTP-capable models without rebuilding locally. The images aim to simplify keeping guides current and provide an easy switch for anyone already using llama.cpp Docker containers until official builds add MTP. This matters because it lowers the barrier to testing and deploying MTP-enabled models, speeds experimentation with multimodal features, and helps standardize developer environments across machines and CI workflows.

Users report that Qwen 3.6—an LLM model—abruptly stops generating outputs mid-response in local deployments. The Reddit thread highlights reproducible cutoffs across prompts and sessions, with community troubleshooting pointing to possible issues in model runtime, tokenization limits, or the hosting framework rather than prompt content. Contributors mention checking inference servers, batching, context-window handling, and decoder settings; some suspect bugs in the model binary or the local inference backend. This matters because abrupt truncation undermines developer trust and production reliability for teams using Qwen 3.6 for apps, chatbots, or pipelines, and it may force rollbacks or workarounds until a fix or patch is released.