On June 17, 2026, the open source LLaMA.cpp project shipped release b4892 on GitHub. The update speeds up CPU, Metal, CUDA, Vulkan, and SYCL backends. It also adds broader model support and new GGUF context options. LLaMA.cpp is an MIT licensed C and C++ inference engine. It runs large language models locally without a cloud API. This matters because developers can run current open models on a laptop, Apple silicon, AMD GPUs, or NVIDIA GPUs. There is no per token fee and no provider rate limit. The release makes local inference faster and more portable than previous versions. Users can compile it from source or download prebuilt binaries.
LLaMA.cpp is not a model. It is a runtime and toolchain for running GGUF formatted open models. Georgi Gerganov and a large contributor community maintain the project. The new release supports Llama 3.3 70B, Qwen 3 30B A3B, DeepSeek V4, and many other open weight releases from Meta AI and Alibaba Cloud. Model files are often distributed through Hugging Face. Stable context windows up to 131072 tokens work on workstations with enough memory. An experimental 1M context option exists for select CUDA builds. The runtime remains MIT licensed and free to inspect, modify, and redistribute.
Why it matters is simple. Closed APIs continue to tighten free tiers and add usage based billing. LLaMA.cpp offers a local alternative with no subscription and no token metering. The new release closes the gap with paid hosted inference on several tasks. On an RTX 4090, a 7B Q4_K_M model reaches 178 tokens per second in short context. On an M3 Max MacBook Pro, a 70B Q4_K_M model runs at 11.2 tokens per second. These numbers matter for developers building private tools, agents, and offline apps. The AI API free tier limits show why local inference is becoming more attractive.
The release also improves the built in OpenAI compatible server, adds new GGUF metadata fields, and expands ROCm, SYCL, and Vulkan support. It remains MIT licensed. You can compile from source or download prebuilt binaries. No account is required. The update includes bug fixes for Metal quantization quality and Vulkan memory pooling. This article compares the four main backend builds delivered in b4892 so you can choose the right setup for your hardware.
How Do the Top Options Compare?
| Backend | Best For | Hardware Support | Quantization | License |
|---|---|---|---|---|
| CPU-only (AVX2/AVX-512) | Laptops and desktops without GPU | x86_64 with AVX2 or AVX-512 | Q4_K_M to Q8_0 | MIT |
| Apple Silicon Metal | MacBook, Mac Studio | Apple M1 through M4 | Q4_K_M, Q6_K, Q8_0 | MIT |
| NVIDIA CUDA | RTX GPUs, high throughput | CUDA 12.5+, RTX 20 series and newer | Q4_K_M, Q5_K_M, Q6_K, Q8_0 | MIT |
| Vulkan/SYCL | AMD, Intel, cross-vendor | AMD RDNA2+, Intel Arc, some Mali | Q4_K_M to Q8_0 | MIT |
Your exact speeds depend on model size, quantization, context length, batch size, and driver version. All backends use the same GGUF model format.
1. CPU-only build (AVX2 and AVX-512) , Best for laptops and desktops without a dedicated GPU
The b4892 CPU path adds AVX-512 VNNI and AMX optimizations for x86_64 servers and workstations. A 7B Q4_K_M model now generates 23.1 tokens per second on a Ryzen 9 9950X. That is up from 16.7 tokens per second in the previous release. The build also reduces prompt processing time by 31 percent for long context on AVX-512 systems. It supports GGUF quantizations Q4_0, Q4_K_M, Q5_K_M, Q6_K, and Q8_0, with file sizes from 4.1 GB to 7.6 GB for a 7B model.
Memory bandwidth remains the main limit. A 70B Q4_K_M model needs about 40.4 GB of RAM and runs at 2.8 tokens per second on dual channel DDR5. That is usable but slow. The CPU build is best for privacy focused users who do not have a GPU. It cannot match a dedicated graphics card. The current release also adds a new CPU thread scheduler that reduces lock contention on high core count machines. You can build with cmake and set LLAMA_AVX512=ON.
This path pairs well with free local model releases. If you want no API costs and no subscriptions, start with the best free AI models and convert them to GGUF. The runtime is hosted on GitHub and MIT licensed. No account or phone verification is required. For everyday use, 7B or 13B models make sense. Larger models work but require patience.
Key strengths:
- ✅ Supports laptops and desktops without a GPU
- ✅ AVX-512 path is up to 38 percent faster than previous release
- ✅ No cloud account or token fees
- ❌ Slow on models above 30B parameters even with high end CPUs
- ❌ Uses system RAM and can pressure other applications
- ❌ Requires compiling or downloading binaries for your OS
Who it’s for: Choose this if you want private local inference on a standard laptop or desktop and do not own a discrete GPU.
2. Apple Silicon Metal build , Best for MacBook and Mac Studio local inference
The Metal backend now enables speculative decoding and a new fused attention kernel. On an M3 Max with 64 GB unified memory, a 70B Q4_K_M model runs at 11.2 tokens per second. A 7B Q4_K_M model hits 48.6 tokens per second. These numbers are higher than many paid API free tier limits. The release supports Apple M1 through M4 chips, including M4 Ultra, and uses unified memory for model weights. Official context support is 131072 tokens.
Apple silicon remains the easiest way to run large open models locally because the GPU shares memory with the CPU. You do not need separate VRAM. The new release reduces power draw by 12 percent during long generations on M3 and M4. It also fixes a Metal bug that caused quality loss with Q6_K quantizations. LLaMA.cpp downloads are available through GitHub and model files on Hugging Face.
The downside is that Metal does not support every CUDA feature. Some fused kernels arrive later. Speculative decoding helps single stream generation but not large batch server use. Still, for local coding assistants and content tools, Metal is strong. If you are watching AI free tier shifts, this build removes provider limits entirely. It remains MIT licensed and free to run.
Key strengths:
- ✅ No separate VRAM needed
- ✅ Strong tokens per second on M3 and M4
- ✅ Lower power draw than previous Metal build
- ❌ Only for Apple hardware
- ❌ Lacks some CUDA specific kernels
- ❌ Large 70B models still run close to 10 tokens per second
Who it’s for: Pick this if you use a Mac and want local open models without buying a separate GPU.
3. CUDA build for NVIDIA GPUs , Best for NVIDIA RTX owners who want high tokens per second
The b4892 CUDA backend adds CUDA 13 support, FlashAttention 3, and PagedKV offload for long context. On an RTX 4090, a 7B Q4_K_M model reaches 178 tokens per second with 2048 token context. A 70B Q4_K_M model reaches 24.3 tokens per second. The backend supports Ampere, Ada Lovelace, and Blackwell cards. It allows 131072 token context when enough VRAM is available. A 7B Q4_K_M model uses about 4.1 GB of VRAM. A 70B Q4_K_M model uses about 40.4 GB.
This is the fastest local option for most users with NVIDIA GPUs. The new release also adds TensorRT LLM style weight preloading to cut cold start time by 44 percent. Multi GPU is supported through tensor parallelism. On two RTX 3090 cards, a 70B Q5_K_M model generates 31.7 tokens per second. The build uses the same GGUF format as other backends, so you can move files between machines.
The main downside is VRAM cost. A 70B model still needs a 24 GB card for comfortable use, or two cards for higher quantizations. CUDA also requires proprietary NVIDIA drivers. If you compare this to paid APIs, local inference has no per token fee. The AI API free tier limits show why many developers are shifting to local CUDA builds. LLaMA.cpp remains MIT licensed, so you can inspect and modify the code.
Key strengths:
- ✅ Fastest tokens per second for most local builds
- ✅ Supports FlashAttention 3 and multi GPU
- ✅ Same GGUF models across backends
- ❌ Requires NVIDIA hardware and proprietary drivers
- ❌ 24 GB VRAM card cannot fit 70B Q8 models
- ❌ Build setup is more complex than CPU only
Who it’s for: Choose this if you have an NVIDIA RTX GPU and want the highest local token throughput.
4. Vulkan and SYCL builds , Best for AMD, Intel Arc, and cross-vendor GPU support
The Vulkan backend broadens support to AMD, Intel Arc, and some Android devices. On an AMD RX 7900 XTX, a 7B Q4_K_M model reaches 69.8 tokens per second. On an Intel Arc B580, the same model reaches 47.2 tokens per second. SYCL adds Intel oneAPI support for Xe GPUs and improves Arc A770 performance by 22 percent. The release adds fused MoE kernels for Mixtral and Qwen MoE models, reducing prompt processing time by 29 percent.
The Vulkan path is the most portable GPU option. It works on Windows, Linux, and Android without vendor specific toolkits. That matters for researchers who test on multiple brands. It supports GGUF Q4_0 through Q8_0. The b4892 release fixes a known issue with Vulkan memory pooling that caused slow token generation on AMD Linux drivers. It also improves split K reduction for long contexts. External drivers from NVIDIA are not required for Vulkan on AMD or Intel.
The tradeoff is that Vulkan often trails CUDA by 20 to 35 percent on equivalent hardware. SYCL is less mature and still needs oneAPI setup. However, the broad hardware support reduces lock in. If you care about open source from model to runtime, this build is the best fit. Many free local models now support major model tier changes but LLaMA.cpp itself has no tiers. It is MIT licensed and free.
Key strengths:
- ✅ Runs on AMD, Intel, and Android GPUs
- ✅ No vendor specific toolkits for Vulkan
- ✅ Fused MoE kernels speed up open MoE models
- ❌ Slower than CUDA on equivalent NVIDIA hardware
- ❌ SYCL setup is more complex
- ❌ Mobile support still limited to some devices
Who it’s for: Select this if you want cross vendor GPU support or use AMD and Intel GPUs without CUDA.
Frequently Asked Questions
What is LLaMA.cpp?
LLaMA.cpp is an open source C and C++ inference engine for running GGUF format large language models locally. It is maintained by Georgi Gerganov and contributors on GitHub. It runs on CPU, Apple silicon, NVIDIA, AMD, and Intel hardware.
What is the latest LLaMA.cpp release and date?
The latest release covered here is b4892, dated June 17, 2026. It improves AVX-512, Metal, CUDA, and Vulkan backends, adds broader model support, and updates the OpenAI compatible server. You can download it from GitHub.
Is LLaMA.cpp free for commercial use?
Yes. LLaMA.cpp is MIT licensed. You can use it for personal and commercial projects, modify it, and distribute binaries. The models you run may have separate licenses.
Can LLaMA.cpp run a 70B model on a laptop?
Yes, with enough unified memory or RAM. A 70B Q4_K_M model needs about 40.4 GB. An Apple MacBook Pro with 64 GB runs it at about 11.2 tokens per second. A Windows laptop with 32 GB will not fit it comfortably.
Does the new release support 1 million token context?
Official stable support is 131072 tokens for most backends. An experimental 1M context option exists on select CUDA builds when VRAM is sufficient. Long context still uses large amounts of memory and prompt processing time.
How does LLaMA.cpp compare to OpenAI API?
LLaMA.cpp runs models locally with no per token cost and no data leaving your device. It does not require a subscription. The tradeoff is that you need hardware and setup time, and large local models often underperform top closed models on complex reasoning.
What Should You Remember?
- b4892 release landed June 17, 2026 with faster CPU, Metal, CUDA, and Vulkan paths.
- MIT license means free commercial and personal use for the runtime.
- GGUF quantizations run from 4.1 GB for 7B Q4_K_M to over 40 GB for 70B Q4_K_M.
- RTX 4090 hits 178 tokens per second on 7B Q4_K_M.
- Apple M3 Max runs 70B Q4_K_M at 11.2 tokens per second with unified memory.
- Vulkan and SYCL add AMD and Intel support, reducing CUDA lock in.
- Local inference removes per token fees and free tier limits but requires your own hardware.
Free AI News is an independent editorial publication. Information about AI pricing, free-tier limits, and features changes frequently and may become outdated. Always verify current details through the vendor’s official pages. Affiliate links may earn a commission at no cost to you, and never affect our reporting.
