Sizing the box
VRAM, bandwidth, and the napkin maths that comes before the invoice
1Overview
A sizing method: turn a model card and a GPU spec into a yes/no answer and a tokens-per-second estimate, before spending anything.
Most hardware advice for local AI is a list of products. This chapter is the arithmetic underneath it: parameters times bit-width plus context gives you the memory you need, and memory bandwidth divided by model size gives you roughly the tokens per second you will get. → Two numbers, and they answer the question every buying discussion actually turns on. → It also carries the awkward 2026 fact that makes "wait for the next generation" the wrong advice this year.
Two pieces of arithmetic that settle most hardware arguments: what fits, and how fast it will run. Parameters × bit-width + context gives memory; bandwidth ÷ model size gives throughput. Everything else on a spec sheet is detail.
Before buying, renting or recommending any hardware for local models — and every time somebody says a model "should run fine".
Parameter count, bit-width, context length, KV cache size, memory capacity and memory bandwidth. Six numbers.
The method is free; the point of it is that the hardware is not. A wrong estimate here is a four-figure mistake.
Bandwidth, not compute, is the ceiling for single-stream generation. Two cards with similar headline TFLOPS and different bandwidth are not similar machines.
2Lessons 8
2.1 Determine if a GPU can run a model
A memory‑capacity check that compares the total bytes needed for weights, KV cache and overhead against the GPU’s available VRAM before assessing speed.
Decide whether a specific GPU will run a model by calculating memory fit and speed
- Identify the model's parameter count and bytes‑per‑parameter
- Calculate required memory as parameters × bytes‑per‑parameter plus KV cache and overhead
- Compare the calculated memory with the GPU's VRAM to confirm it fits
- Determine bytes read per generated token and divide the GPU's memory bandwidth by that value
- Use the resulting token‑per‑second figure to assess whether the speed meets your needs
- You'll see A clear answer showing the memory calculation that fits and the bandwidth calculation that gives the token‑per‑second rate
- Takeaway Check both memory fit and bandwidth, in that order, before buying any hardware
- Check How can you decide if a specific GPU will run a model by calculating both the required VRAM and the token‑per‑second limit from its memory bandwidth?
- Cost Free — both formulas are arithmetic you can do on a phone. The cost of skipping this is buying hardware sized for one of the two questions while ignoring the other.
2.2 Determine model weight memory
Model weight memory, expressed as the product of the total parameter count and the bytes required for each parameter at the chosen precision.
Calculate the weight memory required for a model at various bit‑widths
- Calculate weight memory using parameters × bytes‑per‑parameter
- Multiply 8 billion parameters by 2 bytes to obtain the fp16 size
- Multiply 8 billion parameters by 0.5 bytes to obtain the Q4 size
- You'll see The same 8B model listed at three sizes on a repository – roughly 16 GB (fp16), 8 GB (Q8), 4 GB (Q4) – forming a straight line with bytes‑per‑parameter.
- Takeaway Weight memory equals parameters multiplied by bytes‑per‑parameter, but it is only part of the total GPU budget
- Check What steps let you compute a model’s weight memory at different bit‑widths using the parameters‑times‑bytes‑per‑parameter formula?
- Cost Free — this is arithmetic on a spec you already have (parameter count is on every model card). Getting it wrong costs a GPU bought a size too small.
2.3 Estimate KV‑cache memory for a given context length
The KV‑cache size formula: 2 × layers × kv_heads × head_dim × context_length × bytes_per_element, which yields the cache memory in bytes.
Calculate the KV‑cache size for any model and context using the exact formula
- Identify the model parameters: layers, kv_heads and head_dim
- Select the element size (bytes per element) for the desired precision
- Plug the values into KV cache bytes = 2 × layers × kv_heads × head_dim × context_length × bytes_per_element and compute
- You'll see A numeric value (in bytes or gigabytes) that grows linearly with the chosen context length
- Takeaway KV cache scales with context length, not model size, so it must be included in every memory budget
- Check How do you calculate the KV‑cache size for any model given its layers, kv_heads, head_dim, context length and element size?
- Cost Free — arithmetic, not a benchmark run. The cost of skipping it is an out-of-memory error that only appears once a real conversation gets long, well after the initial "it loaded fine" test passed.
2.4 Estimate single‑stream generation speed using memory bandwidth
A speed ceiling derived by dividing the GPU’s memory bandwidth by the model’s bytes‑read‑per‑token, giving the highest possible tokens per second.
Calculate the theoretical token‑per‑second limit for a chat model on your GPU
- Read the GPU's memory bandwidth from its specifications
- Divide that bandwidth by the model’s bytes read per token to compute a theoretical tok/s value
- Compare the calculated ceiling with real‑world benchmark numbers to gauge the gap between theory and practice
- You'll see An estimated ceiling of around 448 tokens/s for an RTX 5090 running Q4 Llama 3 8B, and a lower measured throughput in practice
- Takeaway Memory bandwidth, not TFLOPS, sets the speed limit for single‑stream generation
- Check What calculation converts a GPU’s memory bandwidth into a theoretical maximum token‑per‑second rate for single‑stream generation?
- Cost Free to compute the ceiling. The cost of ignoring it is choosing a GPU on its TFLOPS number and being surprised that a "faster" card generates chat tokens no quicker than a slower one with the same bandwidth.
2.5 Determine the key GPU specs for local inference
The memory capacity and memory bandwidth specifications of a GPU, which are used respectively for fitting the model and estimating its token‑per‑second ceiling.
Extract memory capacity and bandwidth from a GPU spec sheet and decide which card can run your model.
- Open the GPU specification page for the target card.
- Locate the memory capacity field on the page.
- Locate the memory bandwidth field on the page.
- Apply the memory‑capacity formula to see if your model fits.
- Apply the bandwidth formula to estimate the maximum token‑per‑second ceiling.
- You'll see Two spec pages displayed side‑by‑side with only the memory capacity and memory bandwidth fields highlighted, showing which card fits the model and its theoretical speed.
- Takeaway Memory capacity tells you what fits and memory bandwidth tells you how fast it runs for local inference
- Check Which fields on a GPU spec page must you locate and use to determine whether the card can fit your model and how fast it can generate tokens?
- Cost Free — five minutes reading a datasheet instead of a marketing page. The cost of skipping it is buying on the number the page made biggest, which is rarely the number this workload cares about.
2.6 Measure model throughput before purchasing
The llama‑bench throughput measurement, which prints prompt‑processing and text‑generation columns in tokens per second for the selected model file.
Obtain a real‑world tokens‑per‑second figure for the candidate model on your hardware
- Open a terminal on the machine that will run the model
- Execute llama-bench with the target GGUF file and desired batch size, e.g.
llama‑bench -m model.gguf -b 8 - Observe the printed table containing the pp (prompt processing) and tg (text generation) columns in tokens / sec
- If you need concurrent performance, run guidellm against a live vLLM endpoint with a simulated user load, e.g.
guidellm --model model.gguf --users 10
- You'll see A table printed by llama-bench showing pp and tg columns in tokens / sec, positioned below the theoretical ceiling from earlier lessons
- Takeaway Running llama‑bench (single‑stream) or guidellm (concurrent load) converts a theoretical bandwidth limit into an actual measurement you can trust when deciding to buy
- Check How does running llama‑bench (or guidellm) on your hardware provide a real‑world tokens‑per‑second figure for a candidate model?
- Cost Minutes per run once the tool is installed and the model is downloaded. Free if you already own the hardware; the cost of skipping it is buying based on a number nobody verified on your actual configuration.
2.7 Assess whether waiting for the next generation is realistic
A hardware release calendar lookup that matches the product’s announced launch date against the timeframe suggested by “wait for next gen” advice.
Determine if the advice to wait for new hardware applies to the current release calendar
- Identify the hardware category you are considering
- Search for the official release calendar of that category for the current year
- Compare the dates in the calendar with the timeframe suggested by the "wait for next gen" advice
- Note whether a new product is scheduled before your intended purchase date
- Decide to buy now or wait based on the verified timeline
- You'll see A forum thread from early 2026 still urging patience, while the verified launch schedule shows the next NVIDIA cards arriving at CES 2027
- Takeaway Advice must be checked against actual product timelines rather than assumed
- Check What process lets you decide if waiting for next‑gen hardware is realistic by comparing release calendar dates with your intended purchase timeline?
- Cost Free — one search to check a release calendar before repeating advice about it. The cost of skipping it is delaying a purchase by a year while believing the wait is a few months.
2.8 Size a GPU memory box for your own model
A worksheet that aggregates parameter count, bit‑width weight memory, context‑dependent KV cache, an overhead buffer and the GPU’s bandwidth to produce a fit/no‑fit decision with a token‑per‑second estimate.
Calculate the memory and bandwidth needs of a chosen workload and decide if a specific GPU can meet them.
- Select the model and note its parameter count.
- Choose a bit‑width and multiply parameters by bytes‑per‑parameter to get weight memory.
- Enter a realistic context length and calculate KV cache using the model’s layer, head, and dimension numbers.
- Add a 10‑20 % overhead margin and compare the total against the GPU’s memory capacity.
- Divide the GPU’s memory bandwidth by the weight‑memory figure to obtain a theoretical token‑per‑second ceiling.
- You'll see A completed worksheet showing model details, memory calculations, GPU capacity comparison, and a yes/no decision with a token‑per‑second estimate.
- Takeaway Compute both memory fit and bandwidth ceiling yourself before any price consideration
- Check How do you combine weight memory, KV‑cache size, overhead margin and bandwidth calculations to determine whether a chosen GPU can handle your model workload?
- Cost About an hour end to end: the arithmetic is minutes, the benchmark run is minutes, and the rest is deciding what "realistic context length" means for how you actually use the model.
3See also
💬 Discuss this chapter
Ask, share, or report — over on the Heidelberg AI community forum.