Big memory, slow bandwidth
DGX Spark, Strix Halo and Mac Studio — when 128 GB beats 32 GB, and when it loses
1Overview
Workstations and mini-machines with 96–512 GB of unified memory shared between CPU and accelerator, aimed at running large models locally.
A new class of machine appeared for local AI: unified memory in the hundreds of gigabytes, at a fraction of a discrete GPU's bandwidth. They run models a 5090 cannot load, and they run them slowly. → Which of those two facts matters depends entirely on what you are doing, and this chapter gives you the test. → It carries one number that reframes the whole category: two of the most-recommended boxes of 2026 sit at 273 GB/s, roughly six and a half times slower than a consumer flagship.
A whole product category built on a trade you are rarely shown: enormous memory, modest bandwidth. These boxes load models a flagship GPU cannot and generate tokens far more slowly. The chapter is the decision procedure for telling which half of that sentence applies to you.
Large models, batch or single-user work, and privacy requirements — where "it runs at all" beats "it runs fast".
Unified memory capacity, memory bandwidth, the accelerator, and the offload runtime that makes mixture-of-experts models practical.
The runtimes are free and open; the machines are the expensive part, which is why the comparison method matters more than any single recommendation.
Two widely recommended 2026 machines sit at 273 GB/s — about 6.6× below a consumer flagship. They win on model size and lose on latency, and marketing rarely puts those side by side.
2Lessons 7
2.1 Assess unified memory versus discrete VRAM for large models
Unified memory is the shared pool of RAM accessible by CPU, GPU and NPU on machines such as DGX Spark, Mac Studio and Strix Halo.
Determine whether a model fits in memory on unified‑memory machines compared with discrete GPUs
- Identify the total unified‑memory capacity of each machine (e.g., 128 GB on DGX Spark, Mac Studio, Strix Halo)
- Calculate the memory required for the target model at the chosen quantisation (≈35‑40 GB for a 70B model in 4‑bit)
- Compare the required amount with the available capacity on each system to see if it fits
- You'll see The 70B GGUF file loads on a 128 GB DGX Spark or Mac Studio but fails on a 32 GB RTX 5090
- Takeaway Unified memory lets CPU, GPU and NPU share capacity so fitting large models becomes a matter of total pool size rather than separate VRAM limits
- Check How do you determine whether a 70B model at 4‑bit quantisation fits within the unified memory of each machine?
- Cost Nothing to reason about yet — this lesson is definitional. The cost question is the next one: capacity solved, at what speed.
2.2 Compare memory bandwidth of three big‑memory machines
Memory bandwidth is the rate at which a system can move data between its memory subsystem and processing units, measured in GB/s.
Identify the memory‑bandwidth differences between DGX Spark, Strix Halo and Mac Studio
- Open the NVIDIA DGX Spark specification page nvidia.com/en-us/products/workstations/dgx-spark/ and record the listed bandwidth
- Open the Apple Mac Studio M3 Ultra specifications page apple.com/mac-studio/specs/ and note the unified‑memory bandwidth
- Open the AMD Strix Halo information source, read the theoretical LPDDR5X‑8000 bandwidth and the measured figure, then record both values
- You'll see A table showing each machine’s memory capacity next to its measured or rated bandwidth
- Takeaway Capacity clusters while bandwidth varies widely, revealing that machines marketed together can behave very differently
- Check What steps are required to record and compare the listed memory‑bandwidth values for DGX Spark, Strix Halo and Mac Studio?
- Cost Free to read; the cost is what happens if you skip this step and buy on capacity alone.
2.3 Estimate memory and throughput for a model on a unified-memory machine
The sizing formula multiplies model parameters by bytes‑per‑parameter, adds KV cache size, and divides memory bandwidth by active weight bytes per token.
Apply the sizing formulas to determine if a model fits and how fast it will run on a specific unified‑memory system
- Calculate memory needed by multiplying parameters by bytes‑per‑parameter and adding an estimated KV cache size
- Compare the result to the machine’s 128 GB unified memory to confirm it fits
- Divide the system’s memory bandwidth by the active weight bytes per token (≈35 GB) to estimate tokens per second
- You'll see Two token‑per‑second estimates – one for DGX Spark and one for Mac Studio – derived from the same weight‑size arithmetic
- Takeaway Capacity is rarely the limit with 128 GB, but bandwidth determines the relative speed of different machines
- Check How do you use the sizing formulas to estimate token‑per‑second throughput for a model on a 128 GB unified‑memory machine?
- Cost Nothing — this is arithmetic you already have from sizing-your-ai-box, applied to new numbers.
2.4 Run a mixture‑of‑experts model with hybrid CPU/GPU offload
The hybrid offload path runs active MoE experts on GPU while keeping the remaining weights in system RAM, enabling inference beyond GPU memory limits.
Execute inference for an MoE model whose total parameters exceed GPU memory by using ktransformers’ CPU/GPU heterogeneous path
- Clone the ktransformers repository from GitHub
- Install the required packages with pip install -r requirements.txt
- Download a supported MoE checkpoint and place it in the model directory
- Launch inference using the hybrid flag, e.g. python -m ktransformers.run --model
--offload hybrid
- You'll see Tokens are generated on a unified‑memory machine while the bulk of the MoE weights reside in system RAM and only the active experts run on the GPU
- Takeaway Hybrid offload lets MoE models sidestep the memory limits that block dense models on the same hardware
- Check Which command launches inference with the hybrid offload flag for a mixture‑of‑experts checkpoint using ktransformers?
- Cost Free and open source (ktransformers, mlx); the cost is engineering time to set up the hybrid split correctly, and the same capacity-then-bandwidth arithmetic from the previous lesson, applied to the active-parameter count instead of the total.
2.5 Decide when low bandwidth is acceptable
Low‑bandwidth suitability is judged by whether the job prioritises total batch throughput over per‑token latency and can run unattended.
Pick workloads where a 273 GB/s system can match or beat an 819 GB/s system
- Review the tokens‑per‑second estimate from lesson 3 to understand the worst‑case latency for single‑stream generation
- Identify batch workloads (e.g., summarising thousands of documents overnight) that care about total throughput rather than per‑token latency
- Determine whether the workload requires a human to wait on each token or can run unattended in the background
- You'll see A batch summarisation job finishes on schedule on a DGX Spark even though its single‑stream token rate looks slow in an interactive test
- Takeaway Bandwidth only hurts when a human is waiting on each token; batch, background and privacy‑constrained jobs can tolerate lower bandwidth
- Check What criteria help you decide that a workload can tolerate the lower 273 GB/s bandwidth of a machine?
- Cost The cost is time, not money, when bandwidth is the constraint you are trading against: a batch job on a slow machine takes longer wall-clock time than the same job on a fast one, but may still cost less than renting or buying faster hardware for a workload that never needed the speed.
2.6 Compare new big‑memory machines using a repeatable checklist
The checklist evaluates capacity, bandwidth, model architecture (dense vs MoE), and workload style (interactive vs batch).
Produce a consistent two‑number comparison for any announced machine without running a benchmark
- Identify the capacity number on the vendor page and verify it meets your model’s memory formula including KV cache
- Locate the bandwidth figure (GB/s) on the spec sheet or review and apply the token‑per‑second estimate from the sizing guide
- Determine whether your target model is dense or a mixture‑of‑experts architecture
- Decide if your intended workload is interactive or batch/background
- You'll see A four‑row checklist showing capacity verdict, bandwidth‑derived token rate, dense‑vs‑MoE status and interactive‑vs‑batch classification for the evaluated machine
- Takeaway Four questions—capacity, bandwidth, model type, workload style—turn any unfamiliar big‑memory system into a comparable data point
- Check Which four checklist items allow you to compare any new big‑memory system without running benchmarks?
- Cost Free — this is a reading method, not a purchase.
2.7 Evaluate the memory‑bandwidth trade‑off for each machine
The trade‑off statement links the amount of memory a system provides to the performance penalty or benefit derived from its memory‑bandwidth rating.
State, in one sentence per machine, what each buys and what it costs, and decide when that trade is the right one to make.
- Read the capacity and bandwidth figures for DGX Spark, Strix Halo and Mac Studio.
- Write a single‑sentence summary for each machine that pairs its memory amount with its effective bandwidth cost.
- Match each summary to the workload type (interactive chat, batch summarisation, etc.) where the trade is appropriate.
- You'll see A closing one‑line verdict per machine – DGX Spark, Strix Halo, Mac Studio – each stating what it buys and what it costs, next to the workload it suits.
- Takeaway Big memory is not one machine and not one trade; read the bandwidth number, match it to whether a human waits on the next token, and the capacity headline stops being the thing that decides the purchase
- Check How do you formulate a one‑sentence summary that pairs each machine’s memory amount with its effective bandwidth cost for a given workload?
- Cost Nothing new — this lesson is judgement applied to the numbers gathered in the six lessons before it.
3See also
💬 Discuss this chapter
Ask, share, or report — over on the Heidelberg AI community forum.