Heidelberg AICurriculum

3 New PCs, One Giant AI Model… This Shouldn’t Work

Alex Ziskind ·2026-06-23 ·2 min read

Summary written by us from the video's transcript. The video, and everything in it, is Alex Ziskind's work.

Learn how three compact Intel NUC 16 Pro mini PCs can be combined to run a massive AI model that exceeds any single unit’s memory, and discover the performance trade‑offs of different clustering strategies.

Takeaways

  • Splitting a large model across multiple mini PCs adds significant token‑level latency because each token must travel over the network, so speed does not improve.
  • The Arc GPU accelerates prompt processing but cannot overcome the memory‑bandwidth limit during token generation.
  • Intel’s NPU draws the least power (≈17 W) yet is slower than the GPU; Llama.cpp outperforms OpenVINO on the same hardware for GPU inference.
  • Replicating a model on each node and load‑balancing requests yields near‑linear throughput gains, unlike split‑model clustering which hurts speed.
  • Thunderbolt’s 20 Gbps links do not solve the memory‑bandwidth bottleneck; faster networking alone won’t accelerate token generation.

Why Mini PCs Matter for Local AI

Mini PCs have become powerful enough to run development environments and local AI models without relying on cloud APIs. The new Intel NUC 16 Pro packs Wi‑Fi 7, Bluetooth 6, dual Thunderbolt, dual Ethernet, upgradeable RAM, and an Arc B390 GPU plus a dedicated NPU, making it a compact yet capable dev box.

Hardware Overview of the NUC 16 Pro

The devices come with Intel Core Ultra CPUs (5‑559 to X9), up to 64 GB RAM, integrated Arc GPU and an on‑die AI accelerator (NPU). They expose two Thunderbolt ports that can be used for high‑speed networking. Prices start around $1,700 for a 32 GB configuration.

Benchmarking CPU, GPU, and NPU

Using Llama.cpp on the Arc GPU doubled prompt‑processing speed from ~1,000 to ~2,200 tokens/s, but token generation stayed at ~46 t/s because it is limited by memory bandwidth. The NPU consumed the least power (≈17 W) yet was slower than the GPU for both stages. OpenVINO enabled inference on the NPU, but Llama.cpp outperformed Intel’s own runtime on the GPU by 2.5× (34 t/s vs 14 t/s).

Clustering for Size vs. Speed

When a model exceeds a single box’s RAM, splitting it across three NUCs (total 192 GB) allowed a 75‑GB Llama 3 70B dense model to run, albeit at only ~1.4 tokens/s. The split‑model approach adds network latency for every token, so even upgrading the interconnect from 2.5 GbE to 20 Gbps Thunderbolt did not improve throughput.

Copy‑Every‑Model Strategy for Throughput

Instead of partitioning a model, placing a full copy on each node and routing independent requests to different machines scales linearly. For a model that fits in one box, three nodes achieved ~500 tokens/s versus 196 t/s on a single machine—a 2.5× speed‑up.

Practical Takeaways for Building AI Clusters

If the goal is to run models larger than any single node’s memory, split‑model clustering works but will be slow; it’s mainly useful for enabling otherwise impossible workloads. For higher throughput on models that fit in RAM, replicate the model on each node and distribute requests. The bottleneck across both approaches remains memory bandwidth, not network speed.