llama.cpp Boosts Local LLMs with MTP and Multimodal Fixes

Tiny-vLLM is an open-source C++ and CUDA LLM inference engine and accompanying course designed to teach and implement a high-performance serving stack inspired by vLLM. The project provides a full inference server that loads Safetensors models (demo uses Llama 3.2 1B Instruct) and implements a complete forward pass (prefill and decode) with all computation done in CUDA. Key features include KV cache, static and continuous batching, online softmax/FlashAttention-like kernels, PagedAttention and a paged KV cache, and numerous CUDA kernel optimizations (RMSNorm, RoPE, GEMM, buffer reuse). The repo doubles as a tutorial that walks through tokenization, embeddings, attention mechanics, and GPU engineering techniques—making it relevant for engineers building efficient LLM serving infrastructure.

A community benchmark comparing quantized runtimes for Qwen3.6-27B showed how different quantization schemes and runtimes affect performance and memory on consumer hardware. Shared on Reddit's LocalLLaMA, contributors tested formats (e.g., 4-bit, 8-bit) across runtimes and provided latency, VRAM usage, and accuracy trade-offs. The tests highlight which quantization methods let the 27B Qwen model run on GPUs with limited memory while preserving useful inference quality. This matters for developers and startups aiming to deploy large language models locally or in cost-sensitive environments, influencing choices of quantization strategy, runtime, and hardware for efficient inference.

A new llama.cpp release (tag b9406) fixes MTP and mmproj build issues and addresses a crash in get_rows / mtmd_helper_decode_image_chunk when using MTP with MoE models and vision (reported for Qwen3.6-35B-A3B). The post announces the b9406 release, says the author is building it and asks users to report test results. This matters to developers and researchers running local inference with GGML/llama.cpp, especially those using multimodal MTP (multithreaded processing) and mixture-of-experts models with vision capabilities, since the patch prevents assertion crashes and improves stability. It signals active maintenance in the ggml/llama.cpp ecosystem important for open-source LLM tooling.

A Reddit user posted a speed benchmark of StepFun 3.7 Flash running on an Apple M5 Max, showing real-world performance of this LLaMA-derived inference tool. The post includes a screenshot and links to the benchmark thread, highlighting throughput and latency metrics on M5 Max hardware. This matters because StepFun is part of the growing ecosystem of local LLM runtimes and optimizers, and M-series Apple silicon is becoming a common platform for on-device model inference. The benchmark helps practitioners compare performance across chips and runtimes, informing deployment choices for local, offline, or privacy-sensitive LLM applications.

A Reddit user posted a benchmark of a locally run LLaMA-family model, sharing performance screenshots and resource usage details. The thread, in r/LocalLLaMA, highlights running open-weight large language models on consumer hardware and compares latency and token throughput across configurations. Participants discuss trade-offs in model quantization, CPU vs GPU inference, memory limits, and toolchains like GGML and llama.cpp that enable efficient local inference. This matters because consumer-accessible LLM runtimes lower barriers to experimentation, raise implications for privacy and offline use cases, and accelerate innovation in developer tooling and model optimization. The post illustrates growing community efforts to democratize model deployment outside cloud providers.

LMStudio added support for Multi-Token-Prediction (MTP) and its release notes advise using an MTP-compatible model. The user asks which models others are using with MTP, specifically seeking recommendations for a Qwen 3.6 variant that supports MTP. This matters because MTP can improve throughput and latency for generation tasks, so choosing an MTP-ready model (or a Qwen fork compiled with MTP support) affects performance and compatibility when running LMStudio. Contributors who have tested LMStudio’s MTP feature or maintain MTP builds of Qwen variants are the most relevant sources of practical guidance.

A contributor added MiniCPM5 tokenizer support to the llama.cpp repository via pull request #23384, enabling users to run the MiniCPM5-1B model and its GGUF build on GGML-based runtimes. The PR links to the MiniCPM5-1B model and MiniCPM5-1B-GGUF on Hugging Face, signaling improved compatibility between openbmb’s Chinese-oriented MiniCPM model and the popular llama.cpp inference stack. This matters because tokenizer support is essential for correctly encoding text for inference, broadening the range of models runnable with lightweight, local GGML tooling and helping developers deploy non-English models more easily. It benefits open-source ML tooling, on-device inference workflows, and cross-model interoperability.

A Reddit post documents a $400 local setup running Qwen 3.6-27B on dual consumer GPUs (RTX 3060/3050), achieving roughly 30–50 tokens/second. The builder shares hardware details, VRAM and swap strategies, and configuration steps to host the LLM locally, emphasizing cost-effectiveness and accessibility compared with cloud-hosted models. This matters because it shows practical, low-cost options for running large open models at home, lowering barriers for developers, researchers, and hobbyists who need inference without cloud fees or data privacy concerns. Key players include the Qwen model and NVIDIA consumer GPUs; the post highlights trade-offs in throughput, model size, and memory management when deploying big models on mainstream hardware.

A developer posted on Reddit seeking feedback on a project to simplify running large language models locally, highlighting pain points in model setup, resource management, and user experience. The post mentions efforts to streamline model download, GPU/CPU configuration, dependency handling, and privacy-preserving local inference—aiming to make local AI accessible to non-experts. Contributors discussed trade-offs around performance, model size (quantization), trust, and compatibility with popular models and runtimes. This matters because easier local AI could shift usage from cloud APIs to on-device inference, impacting cloud providers, data privacy, and developer tooling for model deployment. The discussion surfaces real-world requirements for packaging, docs, and UX to broaden local LLM adoption.

A modified Qwen-3.5 27B model labeled “uncensored heretic” has been released with all 15 MTP (multi-turn prompt) states preserved and retained, offered in multiple weight and quantization formats including safetensors, GGUF, NVFP4, NVFP4 GGUF and GPTQ-Int4. The post—originating on a Reddit community for local LLMs—shares downloads and technical details aimed at users running the model locally with different runtime constraints. This matters because distributing uncensored, locally runnable 27B models in efficient quantized formats lowers barriers for offline use, fine-tuning and research while raising safety, licensing and misuse concerns. Developers and infra operators should weigh capabilities against governance and security risks.

A community release claims an uncensored fork of the Qwen3.5 35B model (labelled A3B heretic) preserving all 785 MTP parameters and offering downloads in safetensors, GGUF, NVFP4, NVFP4-GGUF and GPTQ-INT4 formats. The post (originating on Reddit/LocalLLaMA) highlights native MTP preservation and multiple quantized builds for local inference, implying broader accessibility for offline use and experimentation. This matters to developers and researchers seeking higher-fidelity weight preservation and compact, efficient formats for running large LLMs on consumer or edge hardware, but raises safety, licensing and provenance concerns since “uncensored” forks can bypass vendor safeguards and may violate original model terms. Verify legality, model provenance and safety before use.

A five-person Princeton team announced Conifer, a free open-source local inference runtime optimized for Apple Silicon using Rust and handwritten kernels. They've secured funding and report being production-ready enough that feedback from about 100 users will reveal bugs and tooling needs. The team says Conifer outperforms Llama/MLX on small models and aims to match their performance on larger ones over time. This matters because a performant, open local runtime for Apple hardware can shift inference away from cloud services, improve privacy and latency for developers and end-users, and expand the ecosystem for on-device AI. The project’s open-source stance may accelerate adoption and community-driven improvements.

A Reddit user asked which coding-focused LLM best fits an RTX 3060 12GB GPU and whether anyone has run useful models on that hardware. They also requested recommendations on inference stacks (vLLM, llama.cpp) and quantization strategies to reduce memory and improve performance. This matters for hobbyists and developers aiming to run local code-generation or coding-assistant models on midrange consumer GPUs, balancing model capability against VRAM limits and inference speed. Practical choices will affect tooling (GPU vs CPU runtimes), quantization trade-offs (INT8/4/4-bit), and whether to prefer lightweight specialized models or trimmed general models for on-device coding tasks.

Developers added W8A8 activation quantization to MLX, reducing prefill latency on an Apple M5 Pro from 2.84s to 2.52s. The change quantizes activations to 8-bit while keeping weights at 8-bit, improving memory and compute efficiency during model inference. This optimization matters for local LLM deployments and edge inference because it lowers latency and resource use without major model changes, benefiting developers running MLX on consumer-grade Apple Silicon. The work was shared on the LocalLLaMA subreddit, highlighting practical performance gains and signaling broader interest in mixed quantization techniques for faster, cheaper local inference.

A user with an older 4th-gen i7, 32GB DDR3, and no GPU asked how to install and wrap llama.cpp for Python UI use to run small to mid-size LLMs (Qwen 2B/4B/27B, Gemma 31B) on CPU-only hardware. They want guidance on building/packaging llama.cpp (llamacpp) for Python import, performance expectations, model quantization, and whether to use prebuilt wheels, compile with AVX/SSE optimizations, use GGML quantized model files (q4/q8), or employ smaller models and batching tweaks. This matters because CPU-only deployments need aggressive quantization, optimized builds, and careful model selection to be feasible. Recommended focus: compile for your CPU ISA, use quantized GGML models, prefer smaller (<7B) models for practical latency, and consider remote/colab inference or renting CPU/GPU instances if larger models are required.

A user reports running Qwen 3.6 27B MTP (q4_k_xl) in LM Studio on an NVIDIA 3080 Ti (12 GB VRAM) with 128 GB system RAM and seeing about 4.5 tokens/sec. They ask whether this is the hardware limit and whether any tweaks could improve throughput on their current setup. The post highlights typical real-world constraints: a 27B model's working set exceeds 12 GB VRAM so offloading, quantization, and memory-bandwidth/PCIe bottlenecks matter. Potential levers include using more aggressive quantization, CPU/GPU offload settings in LM Studio, reducing batch size or context length, optimizing kernel libraries (e.g., cuBLAS/cuDNN), or moving to a GPU with larger VRAM or NVLink. The question is relevant to practitioners benchmarking large LLMs on consumer GPUs.

A user on r/LocalLLaMA reported trouble getting llama-bench to work with MTP (speculative decoding), saying configurations that work for llama-server fail for llama-bench. They ask whether llama-bench supports speculative decoding or needs a specific “magic incantation” to enable MTP. This matters to developers and researchers running local LLaMA-family models because speculative decoding (MTP) can dramatically speed sampling; incompatibility would limit benchmarking accuracy and performance tuning. Key players are the LocalLLaMA community, llama-bench, and llama-server; the issue points to either missing feature support in llama-bench or configuration differences that need documentation or tooling fixes.