runtime · Georgi Gerganov + the ggml-org open-source community

llama.cpp

llama.cpp is the C/C++ engine that Ollama itself is built on — Ollama wraps it in a friendly Go layer (model pull, versioning, scheduling) that hides the knobs. This chapter teaches the layer underneath: llama.cpp's own server, for when a lab needs something Ollama's abstraction doesn't expose — a specific quantization, a build flag for an unusual CPU, or a single static binary with zero dependencies you can drop on hardware Ollama doesn't officially support. It reads the GGUF model format, and its stated goal is state-of-the-art performance on 'a wide range of hardware, locally and in the cloud': Apple Metal, x86 (AVX/AVX2/AVX512/AMX), RISC-V, NVIDIA CUDA, AMD HIP, Intel SYCL, Vulkan, and CPU+GPU hybrid — often the only realistic engine on a machine with no supported GPU at all. Its own server exposes the same OpenAI-compatible endpoints (chat, completions, embeddings), so it slots into this stack exactly where vLLM would, just with different trade-offs.

• free
• self-host
• Open source