Development & infrastructureGeorgi Gerganov + the ggml-org open-source communityOpen source
llama.cpp
Decision matrix
llama.cpp runtime decision matrix
Overview
llama.cpp is the open-source engine that runs GGUF models on ordinary laptops, Macs and GPUs; LM Studio, Jan and Ollama build on the same ideas. Use it directly when you want every setting and a lean server with its own web page. Beginners should start with an app on top of it.
Other good fits
Squeezing a model onto modest hardware by choosing the quantisation, context length and GPU offload yourself.
Serving the local model behind llama-swap, Open WebUI or an automation on a lab machine.
+ Pros
Read pros
llama.cpp is the engine many local AI tools are built on. It runs open models stored as GGUF files on a normal CPU, an Apple Silicon Mac, or NVIDIA, AMD and Intel GPUs. It ships as small command-line programs: one to chat in the terminal, one to run a server with an OpenAI-compatible API and a simple built-in web chat page. Because nothing is hidden, you choose the model file, how much of it goes on the GPU, and how long the context is.
Pick it when
You want a local model server with no extra layers, for example on a lab PC that answers requests from an automation.
Your hardware is tight and you want to try smaller quantised files or partial GPU offload yourself.
You want to understand what LM Studio, Jan and Ollama do underneath, so you can fix problems in them.
− Cons
It is a toolkit, not a product. There is no model browser with hardware hints, no automatic updates and no user accounts. The many options are powerful but easy to get wrong, and advice you find online goes stale fast because the project changes quickly. It serves a few users well; it is not designed for heavy multi-user traffic.
Pick something else when
You want an app with a model browser: LM Studio or Jan.
You want model downloads and a background service handled for you: Ollama.
A team server must handle many requests at once on a large GPU: vLLM.
Avoid if
You want model downloads and background serving handled for you; Ollama does that.
Many users need high throughput from a big GPU server; vLLM is built for that.
Requirements
Use a release binary for the operating system or build from the official guide. Choose a compatible model format and quantization; memory fit depends on the model, context, precision, backend and workload, not a universal minimum.
Install llama.cpp using the instructions on llama.app, or download a release binary for your system.
Start a server with the small model from the README: llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF. It downloads the model from Hugging Face on first run.
Open the address the server prints in your browser and use the built-in chat page.
Ask it to rewrite three sentences from a real email in plainer English.
Point one other tool (for example an automation's OpenAI node) at the same address.
It worked if the browser chat answers, the second tool gets a reply from the same server, and the server window shows both requests. A 0.8B model is only for testing the setup; switch to a 4B–8B model for real work.
Integration
The server provides documented API routes and a web interface. The runtime is not a language model: model licence, architecture, prompt handling and the selected weights remain separate choices.
Check the result
Expect an interactive terminal session after download and loading. Test a sample prompt against its stated facts; small models can ignore instructions or produce poor prose.
Links & evidence
Documentation-based · not hands-on · Sep 23, 2026
Rests on the llama.cpp GitHub README, build guide and server documentation, checked 23 September 2026. The curriculum team has not installed or run llama.cpp for this review; the first project is a suggested exercise.
llama.cpp README · Sep 23, 2026 — MIT licence; llama cli / llama serve quick start with Qwen3.5-0.8B-GGUF; CUDA, Metal, HIP, Vulkan, SYCL backends; built-in web UI.
Build guide · Sep 23, 2026 — Building for CPU, Metal, CUDA and other backends.
llama-server reference · Sep 23, 2026 — OpenAI-compatible HTTP routes, web UI and server options.
llama.app install page · Sep 23, 2026 — Official installation instructions linked from the README.
Further review notes & alternatives
Costs and operating effort
Prices checked 23 September 2026: llama.cpp is free, open-source software (MIT licence) with no paid plan. You pay for the machine.
Any recent laptop runs the tiny test model. For useful 7B–8B models, plan on about 16 GB of RAM or a GPU with 8 GB of memory. The effort is higher than with an app: you pick model files and settings yourself, and you update the program by hand (it releases very often). If the server should be reachable from other computers, you have to add authentication and a firewall rule yourself.
Alternatives and trade-offs
Ollama takes most of the decisions for you and adds a model library; you lose some fine control.
LM Studio wraps the same kind of engine in a desktop app with hardware-fit hints.
llama-swap sits in front of several llama.cpp servers and starts the right one for each request, so one GPU can offer many models.
Run GGUF models through a configurable local inference engine.
Tune builds, quantization, and hardware-specific acceleration.
Expose chat, completion, and embedding endpoints through its server.
Georgi Gerganov + the ggml-org open-source community · free · self-host