All LLM models — index

A model's knowledge lives in billions of numbers (weights) normally stored at 16-bit precision. Quantization rounds them to fewer bits — 8, 5 or 4 — so the file is smaller and runs faster, like saving a photo as a smaller JPEG. The trade-off has a 'knee': going to 8-bit is almost lossless, 4-bit costs a small, usually-unnoticeable amount of quality, and below 4-bit quality drops sharply. That is why 4-bit (the 'Q4_K_M' setting in GGUF) is the usual sweet spot: pick Q4_K_M for the biggest model that fits your memory, and step up to Q5 or Q8 only if you have room to spare.

These are containers for a quantized model. GGUF is the standard, used by Ollama and LM Studio on any machine — the safe default. MLX is Apple's format, tuned for Mac (Apple Silicon) and often noticeably faster than GGUF on the same Mac, so prefer it if you are on a Mac. MXFP4 is a 4-bit format that OpenAI's gpt-oss models ship in natively, giving full quality with no separate quantization step. In practice: on a Mac, look for an MLX build; everywhere else, download the GGUF (usually the Q4_K_M file).

A useful rule of thumb at 4-bit (Q4): the memory in gigabytes is roughly the number of parameters in billions. So a 7B–8B model needs about 5–6 GB and runs on a normal laptop; a 70B model needs about 40–48 GB and wants a big GPU or a 64 GB+ Mac; and OpenAI's 120B gpt-oss (in its MXFP4 format) is about 61 GB, fitting a single 80 GB GPU or a 128 GB unified-memory box. Add a few GB on top for the KV cache, which grows with how much context you use. If a model does not fit in your VRAM, it either will not run or runs very slowly.

A Mixture of Experts (MoE) model is split into many specialist sub-networks, but only a few 'fire' for each word generated. So while gpt-oss-120b has 117B total parameters, only about 5.1B are active per token — it runs with the speed of a much smaller model while keeping the knowledge of a big one. Combined with its native 4-bit MXFP4 format (which shrinks the weights to about 61 GB), that is how a 120-billion-parameter model fits and runs on a single 128 GB box. Qwen3.6-35B-A3B uses the same trick: 35B total but only about 3B active per token.

The context window is how much text a model can consider at once — the prompt plus its own answer — measured in tokens (roughly ¾ of a word each). A bigger window lets you drop in more at once: a whole book, a large codebase, or dozens of papers, without first building a retrieval system. A 1-million-token model like Gemini 3.1 Pro can hold all of that together and reason across it. For short chats it makes no difference; it pays off when you genuinely need the model to see a lot of material at the same time. Remember that filling a big context uses more memory (the KV cache) and, on paid APIs, costs more.

GLM (Zhipu AI / Z.ai) and MiniMax are Chinese AI labs shipping frontier-class open-weight models, especially strong at coding and agentic work, at a fraction of US-flagship prices — GLM-5.2 rivals Claude Opus on agent benchmarks for roughly a sixth of the cost, and MiniMax-M2 is cheap enough to leave running on a schedule. You can download the weights or use a cheap API (or OpenRouter). One catch worth learning here: 'open weights' is not the same as 'open licence'. Always check the actual licence — GLM-5.2 is MIT and MiniMax-M2 allows commercial use, but the newer MiniMax-M2.7 quietly switched to a non-commercial licence, so the strongest version can't be used in a paid product without permission. And unless you self-host, your prompts still run on the vendor's servers — keep that in mind for sensitive data.

Jan is a free desktop application that lets you run AI language models (like Llama, Mistral, or Qwen) directly on your own computer, so you can chat with an AI without sending anything to the internet. It is made by Menlo Research and works similarly to ChatGPT, but everything runs locally. You download it once, install it like any app, and use it without a subscription or account.

All three are free tools for running local AI models — the main differences are audience and openness. Ollama is command-line focused and best for developers. LM Studio has a polished interface but its app code is closed-source. Jan sits in the middle: a graphical chat interface resembling ChatGPT and fully open-source, making it a good fit for people who want a familiar UI and want to verify the software. Performance differences are small.

Jan's local models use the GGUF format, a file type designed to package AI models efficiently for consumer hardware. GGUF models come in quantized versions (Q4, Q8, etc.) — lower numbers are smaller and faster but slightly less accurate; higher numbers are larger and more precise. Q4_K_M is a good balance. On Apple Silicon Macs, Jan also supports the MLX format for extra speed.

When you run a local model in Jan, everything happens on your own computer — your conversations are never sent to any server, there is no telemetry by default, and no account is required. After the initial model download, Jan works with no internet connection. Your data is stored in a local folder on your machine and goes nowhere else.

A rough guide: 8 GB RAM runs small models up to about 3B for basic chat; 16 GB handles 7B–8B models well (the sweet spot for quality vs speed on most laptops); 32 GB or more is needed for 13B+ models. Jan's Hub flags if a model likely won't fit, so you can check before downloading.

LM Studio is a free desktop application that lets you download and run AI language models directly on your own computer — no internet needed once a model is downloaded. It provides a friendly graphical interface similar to ChatGPT, but everything runs locally. It works on Mac (Apple Silicon), Windows, and Linux.

Because LM Studio's local server mimics the OpenAI API, many third-party tools connect to it out of the box — for example Obsidian, AnythingLLM, Continue.dev, Open WebUI, and n8n. LM Studio also supports the MCP protocol for connecting AI agents to external data sources.

LM Studio is a graphical desktop app — you click buttons, browse models visually, and chat in a built-in interface, friendlier for beginners with no coding. Ollama is command-line first and better for scripting or automation. Both use the same underlying engine (llama.cpp), so quality and speed are comparable; the difference is interface and workflow. A common pattern is LM Studio for exploring models and Ollama for integrating into code.

Yes, after the initial download. You need internet once to install LM Studio and once to download each model. After that, the app and all your models run entirely offline — useful in settings with restricted internet, or when you want guaranteed privacy.

The number (7B, 13B) refers to billions of parameters — a rough measure of capacity. Bigger models generally give smarter answers but need more RAM and run slower. For most everyday tasks on a typical laptop, a 7B or 8B model at Q4_K_M quantization is a practical starting point. LM Studio's Discover tab warns you if a model is likely too large for your hardware.

Slow output usually means the model is too large for your GPU and is spilling into RAM — try a smaller model or a lower quantization (Q4 instead of Q8). Strange or repetitive output is often the wrong chat template; LM Studio normally auto-detects it, but if it looks broken, check the correct template is selected in the model settings. Out-of-memory errors are solved by a smaller model variant.

Yes. Since mid-2025, LM Studio is free for both personal and workplace/commercial use, so you can run it in a lab or office without a license fee. The models themselves are governed by their own licenses (most popular ones permit research use freely). Organizations needing team-sharing or enterprise controls have optional paid tiers, but they're not required for standard use.

Yes. As of mid-2025, LM Studio is free for both personal use and use at work, with no commercial license required. Optional paid Teams/Enterprise tiers exist for organizations needing private collaboration or SSO, but the core desktop app is free, and the models it runs are free and open-source.

On Mac you need Apple Silicon (M1 or newer) and a recent macOS; Intel Macs are not supported. On Windows you need a 64-bit PC with AVX2 CPU support, at least 16 GB RAM, and ideally some dedicated GPU memory. Linux is also supported. You can run it without a dedicated GPU, but it will be noticeably slower.