llama.cpp Advances Multimodal, MTP Performance

A user reports running large Mixture-of-Experts models—Qwen 3.6 35B-A3B and Gemma 4 26B-A4B—on a $200 secondhand PC (i7-6700, GTX 1080, 32 GB RAM) using llama.cpp with TurboQuant/RotorQuant KV-cache quantization to fit a 128k context in 8 GB VRAM. They claim throughput exceeding 24 tokens/sec and show benchmark tables for Q4_K_M quantized builds, demonstrating multi-expert inference on constrained hardware by offloading and compressing KV cache. This matters because it lowers the hardware barrier for running large-context MoE and dense LLMs, enabling researchers and hobbyists to experiment with massive models and long-context workloads without high-end GPUs or cloud costs. The post highlights practical quantization and engineering tricks rather than new model releases.

Developer released Docker images for llama.cpp that include recent MTP (Multi-Token Programming) PR improvements—notably image support and bug fixes—so users can run MTP-capable models without rebuilding locally. The images aim to simplify keeping guides current and provide an easy switch for anyone already using llama.cpp Docker containers until official builds add MTP. This matters because it lowers the barrier to testing and deploying MTP-enabled models, speeds experimentation with multimodal features, and helps standardize developer environments across machines and CI workflows.

Users report that Qwen 3.6—an LLM model—abruptly stops generating outputs mid-response in local deployments. The Reddit thread highlights reproducible cutoffs across prompts and sessions, with community troubleshooting pointing to possible issues in model runtime, tokenization limits, or the hosting framework rather than prompt content. Contributors mention checking inference servers, batching, context-window handling, and decoder settings; some suspect bugs in the model binary or the local inference backend. This matters because abrupt truncation undermines developer trust and production reliability for teams using Qwen 3.6 for apps, chatbots, or pipelines, and it may force rollbacks or workarounds until a fix or patch is released.

A user reports an error when attempting to run an MTP (Mixture-of-Token-Prompt) model with llama.cpp. They built the mtp-pr branch from source and downloaded an MTP-formatted GGUF model (Qwen3.6-27B-Q6_K) from Hugging Face, but encounter a runtime error when launching the model. This matters because community forks and experimental branches like mtp-pr enable new attention/adapter techniques (MTP) in local inference frameworks such as llama.cpp, affecting developers who want to run advanced pretrained models locally. The report flags pain points in model format compatibility, build/runtime setup, or required runtime flags and helps maintainers and users debug integration between model artifacts on Hugging Face and local inference code.

A community contributor merged a pull request to ggml-org/llama.cpp adding MiMo v2.5 vision support, extending the lightweight C++ inference library used for running LLaMA-family models. The change, contributed by AesSedai and tracked as PR #22883, integrates multimodal vision capabilities into the ggml-based runtime, enabling image-aware inference on local, CPU-focused deployments. This matters because llama.cpp is widely used to run open-weight LLMs on consumer hardware; adding MiMo v2.5 vision lets developers and hobbyists build local multimodal applications without cloud dependencies, improving privacy and lowering cost. The update reflects ongoing community-driven enhancements to open-source model runtimes and local AI tooling.

A developer reports extremely poor performance running llama.cpp models via Vulkan on an Intel Arrow Lake integrated GPU (Arc 130T), measuring about 100 tokens/s for pp256 and under 4 tokens/s for tg64 with Gemma 4 E4B—worse than recent CPUs. They tried Vulkan because it was easier to configure than SYCL but stopped before completing SYCL setup. The user asks whether SYCL yields better performance on Intel iGPUs or if alternative runtimes/frameworks should be used. This matters to developers and researchers trying to run local LLM inference on Intel integrated GPUs, highlighting toolchain, driver, and backend trade-offs for on-device ML acceleration.

A user reports slow input processing when running OpenCode alongside llama-server locally despite decent throughput (~21 tokens/sec with Qwen 3.6) and a machine with 32 GB RAM and a 780M iGPU. They observe available RAM (~8+ GB) in tmux and say the model runs fine once it begins "thinking," but OpenCode still delays on each new input. The post asks what OpenCode is doing during that delay and includes server startup details and a video (truncated in the excerpt). This matters to developers deploying local LLM stacks because UI or orchestration overhead—such as prompt tokenization, context window loading, model warm-up, I/O, or synchronous request handling—can create apparent latency even when raw model throughput is acceptable.

A developer reports a practical workflow for running local LLMs on a MacBook Pro with 24GB RAM, highlighting Qwen 3.5-9B (q4_k_s) as the best-performing model so far: it supports a 128K context window, tool use, and ~40 tokens/sec in LM Studio while leaving headroom for other apps. The author compares runtimes and tooling—Ollama, llama.cpp, and LM Studio—and details configuration tweaks (temperature, top_p, K cache quantization, and enabling a “thinking” mode via a prompt template). They share concrete pi and OpenCode config files for connecting to LM Studio and note trade-offs among models (size vs. usability) and harnesses (Pi vs OpenCode).

OpenAI-style hosted APIs remain smoother than local LLM setups, argues Armin Ronacher, who wants local models to be genuinely usable for everyday coding agents. He praises the progress in runtimes, quantization and engines (llama.cpp, Ollama, vLLM, etc.) but highlights user-experience gaps: complex configuration, poor support for tool-parameter streaming, long inactivity timeouts, and brittle stacks that make local inference feel unfinished. Ronacher calls for focus and polish—streaming tool calls, better defaults, unified interfaces and improved integrations—so local models can be competitive without forcing developers back to hosted services. The piece matters for developers, toolmakers and infra projects aiming to broaden local AI adoption.

A developer reports successful multi-tensor parallelism (MTP) for Qwen3.6-27B running on dual AMD MI50 GPUs, claiming up to 1.5x speedup generally and as much as 2x with tensor parallelism. They sought MTP-compatible Q4_1 quantized weights to boost performance on older cards but couldn’t find them, instead discovering related extracted MTP tensor GGUF resources on a community forum. The note highlights practical gains for running large language models on legacy AMD hardware and underscores the role of community tooling and quant formats in squeezing performance from constrained accelerators. This matters for developers and infra engineers optimizing cost-sensitive local or edge deployments of LLMs.

A developer reported achieving over 80 tokens/sec and 128K-token context support on a 12GB GPU by combining the Qwen-3.6 35B A3B model with llama.cpp and the MTP (multi-precision tiled processing) patch. Using an updated llama.cpp build plus the MTP PR and specific quantization/packing techniques, the poster benchmarks token generation speed on a public script and describes configuration tweaks that fit large models into limited VRAM. This matters because it lowers the hardware barrier for running high-context, large-parameter models locally, enabling researchers and hobbyists to experiment without high-end GPUs and advancing accessible LLM deployment. Key components: Qwen-3.6 35B A3B model, llama.cpp, and the MTP patch.

A user asked when llama.cpp will add official support for multi-threaded processing (MTP) on Vulkan/HIP to ease building on Windows 11 GPUs like the Strix Halo after failing to compile the project using CMake. They report spending hours on build errors and are seeking a release or guidance that would provide native Vulkan/HIP backends with MTP to improve performance and simplify setup. This matters because official MTP-enabled builds or clearer platform support would help developers and hobbyists run large language models locally on consumer GPUs, reducing friction from complex build systems and third-party forks. No official timeline was provided in the post.

A developer reports getting MTP (Memory Tensor Paging) working together with TurboQuant TBQ4_0 (a lossless 4.25 bpv KV-cache quantization) on Qwen3.6-27B, achieving 80–87 tokens/sec and a 262K-token context on a single RTX 4090 after optimization. Initial runs compiled at ~43 t/s; optimizations and MTP draft acceptance (~73%) raised throughput significantly. The work shows practical feasibility of large-context inference for a 27B LLM on consumer GPUs using TBQ4_0 and paging techniques, which matters for lowering hardware requirements and enabling long-context applications. This is relevant to researchers and engineers exploring efficient inference, quantized KV caches, and model runtime engineering.

A developer recounts a failed overnight job on a hosted API and argues for local models as reliable long-running “marathon” engines. After a remote service outage froze a scrape-and-summarize agent, they shifted to running Gemma 4 (31B) locally and found a sweet spot: models that fit on consumer GPUs and run offline without quotas or downtime. The piece contrasts cloud “sprint” uses—high-precision, compute-heavy iterations—with local setups optimized for endurance and continuous work. It highlights recent advances, notably Gemma 4 and Multi-Token Prediction (MTP), which boost local throughput so models can process more tasks overnight without burning out. This matters for developers needing resilient, cost-effective uninterrupted processing.

A developer reports disappointment with Qwen 3.6's coding assistance while migrating from Codex. They are using a midsize stack (Kotlin Android app, Rust backend, Postgres) and have tried feeding well-documented features into a local setup combining llama.cpp, Opencode, and Qwen 3.6 (27B/35B, Q4_K_M, 128K context) with tooling for rules, skills, multi-code projects, and code indexing. The user describes reliability and quality issues: hallucinations, incorrect or non-compilable code, poor handling of medium-complexity tasks, and failure to follow provided constraints. They note occasional useful snippets but overall regression versus expectations, highlighting limits of current large open models for dependable software engineering at scale.

A community implementation of Multi-Token Prediction (MTP) for LLaMA.cpp reportedly speeds up Gemma 4 inference by about 40%. Posted on the LocalLLaMA subreddit, the patch adapts MTP—predicting multiple tokens per pass—to the popular C++ LLaMA runtime, improving throughput without changing model weights. Key players include the LLaMA.cpp project and Gemma 4 model; contributors are community developers sharing code and benchmarks. This matters because MTP boosts performance on CPU and lightweight deployments, lowering latency and compute costs for local AI inference and enabling better UX on edge devices. If adopted upstream, it could become a practical optimization for many open-source LLM runtimes and apps.

A community contributor uploaded a GGUF build of the nvidia/Gemma-4-26B-A4B-NVFP4 large language model and provided a companion Docker image (catlilface/llama.cpp:gemma4_26b_nvfp4) because the main llama.cpp branch currently doesn’t support running it. The author warns limited testing due to only having an NVIDIA RTX 5070 Ti and invites feedback on performance and compatibility. This matters for developers and researchers wanting to run Gemma-4 variants locally or on GPUs, as GGUF is a portable format and the Docker image simplifies setup while compatibility gaps in llama.cpp could affect adoption and reproducibility.

Developers running large local models (e.g., Qwen 3.6) and agent frameworks (Llama.cpp, OpenCode, Pi, Basically all agents) are struggling with context compaction, cache validation, and token/response consistency when stitching multi-turn history across systems. The article details practical setups (model command lines, server ports) and highlights issues around prompt/template propagation, preserving thinking states, and ensuring cache keys reflect real context to avoid stale outputs. It stresses why accurate cache invalidation and deterministic compaction matter for correctness, latency, and safety when agents share histories or rely on partial context. The piece matters because these are core engineering problems for deploying LLM-based agents reliably at scale in research and production.