llama.cpp Boosts Local LLMs with MTP and Multimodal Wins

A user reports running Qwen 3.6 27B MTP (q4_k_xl) in LM Studio on an NVIDIA 3080 Ti (12 GB VRAM) with 128 GB system RAM and seeing about 4.5 tokens/sec. They ask whether this is the hardware limit and whether any tweaks could improve throughput on their current setup. The post highlights typical real-world constraints: a 27B model's working set exceeds 12 GB VRAM so offloading, quantization, and memory-bandwidth/PCIe bottlenecks matter. Potential levers include using more aggressive quantization, CPU/GPU offload settings in LM Studio, reducing batch size or context length, optimizing kernel libraries (e.g., cuBLAS/cuDNN), or moving to a GPU with larger VRAM or NVLink. The question is relevant to practitioners benchmarking large LLMs on consumer GPUs.

A user on r/LocalLLaMA reported trouble getting llama-bench to work with MTP (speculative decoding), saying configurations that work for llama-server fail for llama-bench. They ask whether llama-bench supports speculative decoding or needs a specific “magic incantation” to enable MTP. This matters to developers and researchers running local LLaMA-family models because speculative decoding (MTP) can dramatically speed sampling; incompatibility would limit benchmarking accuracy and performance tuning. Key players are the LocalLLaMA community, llama-bench, and llama-server; the issue points to either missing feature support in llama-bench or configuration differences that need documentation or tooling fixes.

A Reddit post titled “Experts first llama.cpp” highlights a community discussion around llama.cpp, an open-source C/C++ implementation for running LLaMA-family language models locally. Contributors share expertise, tips, configurations and performance trade-offs for various hardware, aiming to help users optimize inference on CPUs and small GPUs. The thread matters because llama.cpp has become a cornerstone tool enabling offline, privacy-preserving access to large language models outside cloud providers, lowering barriers for developers, researchers and hobbyists. Practical advice in the discussion can speed adoption, improve efficiency, and influence how local AI tooling evolves across edge devices and self-hosted setups.

Llama.cpp users report that asymmetric KV cache quantization (e.g., -ctk q8_0 -ctv q4_0) forces prompt processing onto CPU for CUDA builds, drastically reducing PPS and performance. The discussion in the GGML/llama.cpp repo highlights current caveats: mixed quant modes can break GPU execution paths, impacting latency and throughput, and proposed code adjustments or workarounds are being debated to preserve GPU-side KV caching. This matters to developers and deployers of on-device and server LLM inference because quantized models aim to reduce memory while keeping speed; ensuring GPU-compatible KV cache handling is critical for practical low-cost, high-performance inference. Contributors and maintainers are exploring fixes and trade-offs.

A user asks which to choose between an unnamed Strix Halo with 128GB and an M5 Pro with 64GB (or comparably priced MacBook Pro 16" / Mini PC around $2,500–$3,000) for AI workloads. They mention using LM Studio and prefer macOS for DrawThings over ComfyUI, noting differences in GPU-available RAM—48GB vs 96GB—affecting model performance. The decision hinges on RAM capacity, platform/tooling compatibility, and workflow convenience: macOS offers friendlier GUI tooling, while higher GPU RAM on other hardware can enable larger models and faster inference. Buyers should weigh software ecosystem, model support, and real-world benchmarks for their specific ML tasks.

A pull request to the ggml-org/llama.cpp repository (PR #22929) fixes repeated prompt processing that affected users running llama.cpp with OpenCode and Pi integrations. The change stops unnecessary reprocessing of prompts, improving efficiency and performance for local or embedded LLM workloads using llama.cpp as the inference engine. This matters because OpenCode/Pi users often deploy llama.cpp for on-device or low-resource inference; reducing redundant prompt handling lowers latency, CPU usage, and power draw, and improves real-time interaction quality. The PR is linked for review and testing; maintainers and downstream projects should evaluate and merge to propagate the fix to clients and distributions.

A developer released LlamaStation v0.9, a Windows GUI front end for running local LLMs via llama.cpp that adds multi-backend support (including GGML backends), TurboQuant quantization, MTP (multi-turn processing), and other convenience features. Built as a side project with AI assistance, LlamaStation targets users who prefer clicking over command-line workflows, offering an easier way to load models, manage quantization, and switch runtimes. It matters because GUIs like this lower the barrier to running local open-source models, broadening access for hobbyists and developers while promoting experimentation with quantization and alternative inference backends. The project is open to contributions and improvements via PRs.

A developer posted a lightweight utility that streamlines searching Hugging Face model repositories, reportedly coded using Qwen 3.6-27B. The tool simplifies finding and filtering models on Hugging Face, improving discovery for local LLM deployments and researchers. Key players include the Hugging Face model hub and the Qwen 3.6-27B large language model used to assist or generate the utility code. This matters because easier model discovery speeds iteration for developers deploying local or custom models, reduces friction for benchmarking and prototyping, and showcases how modern LLMs can bootstrap developer tooling. The post surfaced on a LocalLLaMA subreddit, indicating community interest in tooling that bridges LLMs and model hub ecosystems.

A user on Reddit asked for recommendations on the “best” Qwen 3.5 or 3.6 “reap” (pruned) model for agentic coding, citing performance constraints on a low-VRAM setup. The post links to a specific Hugging Face repository, tvall43/Qwen3.5-14B-A3B-Claude-4.6-Opus-Reasoning-Distilled-reap-gguf, described as a pruned GGUF build that runs about twice as fast for the user. The key concern is whether pruning sacrifices important capabilities needed for agentic coding workflows, such as reasoning quality or tool-use reliability. No benchmarks, dates, or comparative results are provided in the excerpt, and the content is primarily a request for community guidance rather than a reported model release or evaluation.

A user asked how to run Gemma 4 31B with MTP in LlamaCPP after noticing LlamaCPP now requires a combined GGUF that includes both the main model and the MTP drafter, rather than accepting a separate drafter GGUF. They report there is no prebuilt combined main+MTP GGUF available for Gemma 4 31B and seek guidance on using Gemma 4’s MTP capability under the updated LlamaCPP requirements. This matters for developers and hobbyists running local inference: without a combined GGUF they can’t enable MTP in current LlamaCPP builds, so solutions include creating a merged GGUF, using a different runtime that supports separate drafters, or awaiting upstream model packaging or LlamaCPP changes.

KV Cache Is Becoming the Memory Hierarchy of Inference

A recent pull request to the ggml-org/llama.cpp repository (PR #23269) introduces MTP-related improvements for the LLaMA C++ inference engine. The update, shared by a Reddit user, targets performance and/or memory enhancements tied to MTP (mixed-precision / tensor processing) that could speed local LLaMA model inference and efficiency. This matters because llama.cpp is a core open-source runtime used to run LLaMA-family models on local hardware, and efficiency gains directly affect latency, hardware requirements, and energy use for developers and hobbyists. Users running local LLaMA instances should review and merge the changes to benefit from improved throughput and resource use.

Mac users are debating whether to stick with MLX quantized models or switch to GGUF with MTP, as benchmarks for token generation and prompt processing vary. The post notes LM Studio handles MLX poorly due to bad caching and lack of MTP, while omlx offers strong caching, turboquant and dflash but also lacks MTP (which may arrive soon). GGUF gains MTP support that can accelerate multi-threaded processing, potentially improving throughput on macOS, while tooling and cache behaviors across runtimes (LM Studio, omlx) influence real-world performance. The choice matters for mac users optimizing local LLM inference latency and efficiency depending on model format, runtime features, and upcoming MTP support.

The ggml-org/llama.cpp project released version b9200, introducing changes to prompt processing that avoid copying logits for every token in a batch during multi-token processing (MTP). The update — linked on the project's GitHub release page and discussed in a recent comment by user am17an — aims to improve efficiency and could increase tokens-per-second or prompt-processing performance (pp) for batched inference. This matters to developers and researchers running LLaMA-family models locally or on resource-constrained hardware because reduced memory operations can lower latency and CPU/GPU overhead. The release signals incremental optimization in popular open-source ML runtimes, benefiting deployment and experimentation.

A hands-on test shows running large models locally may not be cheaper than using cloud inference. Williamangel benchmarked Gemma 4 31B on a 14" M5 Max MacBook (64GB, $4,299) and observed 10–40 token/s local throughput; accounting for electricity ($0.18–$0.20/kWh) yields $0.40–$4.79 per million tokens. By contrast, OpenRouter cloud instances serving the same model cost $0.38–$0.50 per million tokens and some providers deliver 60–70 token/s—faster and often cheaper. The analysis argues proponents of local inferencing ignore hardware purchase, depreciation, idle time, and other operational costs; perceived benefits like one‑time payment and privacy drive the myth that local equals cheaper. The piece reframes cost comparisons for AI deployments.

A user reports switching from Qwen 3.6 35B A3B to Qwen 3.6 27B to try the new MTP speculative draft and shares their llama-server launch flags. They list model path, context length, number of layers to keep low-level (ngl), parallelism, thread count, jinja, host/port, reasoning budget, and speculative type set to draft-mtp, but the snippet cuts off before spec-draft-n-max. The post seeks experiences or tuning tips for MTP speculative sampling on a 27B Qwen model—important for practitioners optimizing latency and throughput when running medium-large LLMs locally or on servers. This matters to developers and sysadmins deploying inference pipelines and exploring speculative decoding trade-offs.

A hands-on cost analysis finds running LLMs on an M5 Max MacBook Pro is generally more expensive than using cloud-hosted OpenRouter models. The author compares electricity (~$0.18–$0.20/kWh), hardware amortization for a $4,299 M5 Max laptop over 3–10 years, and measured token throughput (10–40 tokens/sec) to compute per-million-token costs. Depending on lifespan, power draw and throughput, on-device inference ranges roughly $0.40–$4.79 per million tokens, while OpenRouter’s Gemma 4 31b runs about $0.38–$0.50 per million tokens and is often 2x faster. Conclusion: hardware amortization dominates local costs and cloud inference is typically cheaper and faster, though running advanced models locally on consumer silicon is increasingly viable.

Benchmarks posted on Reddit show Qwen 3.6-27B Dense running with MTP (Mixture of Textual Prompts?) on an ASUS Strix Halo Windows setup, reporting performance and inference behavior. The post links to an image with benchmark results, implying plug-and-play testing of the Qwen 27B dense model under Windows using local LLM tooling. This matters to developers and hobbyists deploying large open models locally because it provides real-world performance reference for a recent Qwen variant on consumer-grade hardware and may inform choices about model size, runtime settings, and hardware compatibility. The content is practical for people evaluating local inference latency, memory usage, and setup details for running Qwen models on Windows.