The model on your bench
A 2 GB file that answers questions with your laptop and nothing else
1Overview
A small open-weights model, downloaded once, running in a container on your own machine.
The bench ships with a small model that runs on the CPU of an ordinary laptop — no graphics card, no account, no internet. This chapter is about what such a model is genuinely good at, what it is not, and how to swap it for a different one. Knowing the difference is the difference between "local AI is useless" and using it for the half of the work it does well. → `local-private-models` is the wider landscape of models you can run yourself.
qwen2.5:3b is about 2 GB and answers on a CPU. That is a remarkable place to be: five years ago nothing this capable ran on a laptop at all. It is also genuinely limited, and the honest lesson is where the line falls. Good: summarising, extracting fields, rewriting, classifying, drafting, and every exercise in this course that is about mechanics rather than brilliance. Weak: long chains of reasoning, code of any size, anything where a subtle mistake is expensive.
Everything private, everything offline, and everything where a good-enough answer is genuinely good enough.
It answers through the gateway like any other provider — nothing about the apps knows it is local.
On 8 GB of RAM, run the model or a browser full of tabs, not both. Stop the model when you are using a cloud provider.
2Lessons 2
2.1 Find the edge of a small model, deliberately
A 3B model has read a lot and can hold little. Testing it on purpose is faster than discovering its limits inside a task you cared about.
Build your own sense of where local stops being enough.
- Give it a summary task with a page of text. It will do well.
- Give it a extraction task — "list every date and what happened on it". Also well.
- Give it a three-step reasoning task — a puzzle, a proof, a refactor. Watch it produce something confident and wrong.
- Now switch to a big model and run the same three. The first two look identical. That is the whole lesson.
- You'll see Two of three tasks where the free local model is indistinguishable from the paid one.
- Takeaway Pick the model per task, not per person. Most course work runs fine on the small one.
2.2 Swap in a different model
The model app on the bench is Ollama. It can hold several models and answer with whichever you name.
Do this first Find the edge of a small model, deliberately
Pull a second model and compare the two on your own task.
docker exec ollama ollama pull llama3.2:1bin a terminal. 1b is smaller and faster than the shipped 3b — a good contrast.
- Pull it. The tile lists every model it holds, with the one currently answering marked.
- Point the bench at it through "Which AI answers" — the bench model option, with the new name.
- Run your own task on both. Speed against quality, measured on the thing you actually do.
- Disk matters. Each model is gigabytes; remove what you do not use with
docker exec ollama ollama rm.
- You'll see Both models on the tile, and a visible speed difference between them.
- Takeaway Smaller is faster and dumber, and where the line falls depends on your task — so test it on your task.
💬 Discuss this chapter
Ask, share, or report — over on the Heidelberg AI community forum.