Heidelberg AICurriculum
Track 16 · Advanced
16.4

Beyond NVIDIA

The NPU already in the laptop you were issued

7 lessons 2026-08-13 AI-generated

1Overview

Local inference on non-CUDA hardware: Intel iGPU and NPU, AMD Ryzen AI, and the Windows on-device stack.

Every local-model chapter in this course quietly assumes a discrete NVIDIA card. For an organisation with a hundred ordinary office laptops and no GPU budget, that assumption is the entire problem. → This chapter runs models on what is already there: the integrated GPU and the neural processing unit inside recent Intel and AMD laptops, through the vendor runtimes built for exactly that. → It is deliberately honest about the ceiling. These machines can do useful work and they cannot do all of it, and knowing which is which is the skill.

1.2After this chapter you can
→Identify the NPU or iGPU in a machine and what it is actually rated for
→Run a small model on an NPU through the appropriate vendor runtime
→Convert a model into the format a non-CUDA runtime expects
→Measure what an office laptop can and cannot do without a discrete GPU
→Pick the right runtime for Intel, AMD and Windows Copilot+ hardware
→Make a procurement argument that does not start with "we need GPUs"
1.3When to reach for it

A fleet of ordinary laptops, no GPU budget, and a privacy or offline requirement that rules out an API.

1.4Key parts

A model converted to the runtime's format, a vendor runtime, a quantization the NPU supports, and a realistic size ceiling.

1.5Free vs paid

Every runtime here is free and open, and the hardware is already bought. That is the entire argument for this chapter.

1.6Watch out

NPU support is model-specific and format-specific. "It runs on the CPU" and "it runs on the NPU" are different claims and only one of them is efficient.

2Lessons 7

2.1 Find the laptop's NPU

The NPU graph appears next to CPU, Memory and GPU in Task Manager’s Performance tab.

Identify whether a laptop has an NPU, which vendor supplies it and its TOPS rating

  1. Open Task Manager and select the Performance tab to look for an NPU graph
  2. Open Device Manager, expand the Neural processors node and note any listed NPU device
  3. Run python -c "import openvino as ov; print(ov.Core().available_devices)" in a command prompt after installing OpenVINO to see if OpenVINO detects an NPU
  4. Execute lemonade backends in a terminal to list AMD‑specific accelerators and confirm the presence of an XDNA NPU
  • You'll see Task Manager’s Performance tab shows an extra graph labelled NPU next to CPU, Memory and GPU on Copilot‑plus laptops; older machines stop at the GPU graph
  • Takeaway Know if a machine is no‑NPU, sub‑40‑TOPS or Copilot‑plus NPU before choosing a runtime
  • Check What steps let you confirm whether the laptop includes an NPU, which vendor supplies it, and its TOPS rating?
  • Cost Free — five minutes per machine, no install required for the Windows check.

2.2 Run an instruct model using the matching vendor runtime

Lemonade, OpenVINO GenAI, and ONNX Runtime GenAI are the runtimes matched to AMD, Intel, and Copilot+ silicon respectively.

Execute a small instruct model with the SDK that aligns to your machine’s silicon

  1. Open Terminal (or Command Prompt on Windows) in a new window
  2. Install the appropriate SDK with pip install lemonade-sdk for AMD Ryzen AI, pip install openvino-dev[genai] for Intel, or pip install onnxruntime-genai for Copilot+ devices
  3. Run the model using the vendor‑specific command: lemonade run model.onnx, python -c "import openvino_genai; pipeline = openvino_genai.LLMPipeline('model.xml'); print(pipeline.generate('Hello'))", or python -c "import onnxruntime_genai as og; gen = og.Generator('model.onnx'); print(gen.generate('Hello'))"
  4. Observe the generated text printed to the console confirming successful execution
  • You'll see The same 1–4B instruct model prints generated text when run with Lemonade on AMD Ryzen AI, OpenVINO GenAI on Intel, or ONNX Runtime GenAI on Windows Copilot+
  • Takeaway Match the runtime to the silicon – Lemonade for AMD, OpenVINO for Intel, ONNX Runtime GenAI for Windows Copilot+ – and treat cross‑vendor support as optional
  • Check Which command installs the correct SDK for your machine’s silicon and how do you verify that the model runs successfully?
  • Cost All three runtimes are free, open-source, Apache-2.0-family licensed. The only cost is the time to learn the one your fleet actually needs.

2.3 Convert a Hugging Face model for on‑device runtime

optimum‑cli export openvino, lemonade pull, and Hugging Face ONNX exports generate runtime‑ready artifacts.

Produce a converted and quantised artifact that your chosen on‑device runtime can load directly

  1. RUN optimum-cli export openvino with the appropriate model and weight format to generate OpenVINO IR files
  2. EXECUTE lemonade pull to retrieve a pre‑converted GGUF, FLM or ONNX artefact from Lemonade’s catalogue
  3. DOWNLOAD a pre‑converted ONNX model such as microsoft/Phi-3-mini-4k-instruct-onnx from Hugging Face for use with ONNX Runtime GenAI
  • You'll see A folder containing an .xml file next to a matching .bin file after the conversion command finishes
  • Takeaway Each NPU runtime requires its own converted, quantised artefact – raw Hugging Face checkpoints cannot be used
  • Check What commands produce a converted and quantised artifact that can be loaded directly by the chosen on‑device runtime?
  • Cost One conversion, once, per model — CPU-bound, a few minutes for a 1–4B model. Skipping it is not an option: none of these three runtimes will load an unconverted checkpoint.

2.4 Generate text on the laptop NPU

The device string "NPU" passed to the OpenVINO pipeline selects the laptop’s neural processor for inference.

Run a tiny instruct model and obtain a response using the NPU runtime on real hardware

  1. OPEN Terminal and install the OpenVINO packages with pip install -U openvino openvino-genai
  2. TYPE the four‑line Python script that imports the library, creates a pipeline with the model folder and the device string "NPU", then prints the generated text
  3. RUN the script; observe the first request take longer (cold compilation) and subsequent requests return quickly
  • You'll see A short sentence or two appears as the model’s reply, with the NPU utilisation graph moving in Task Manager or the backend name shown in the console
  • Takeaway Switching to the NPU only requires changing a single identifier or using the appropriate CLI, because all heavy lifting was done during identification and conversion
  • Check How does changing the device identifier in the Python script enable running the instruct model on the laptop’s NPU?
  • Cost Free — no API call, no token bill. The cost already happened: it is the laptop you were issued.

2.5 Measure real‑world token throughput on the laptop NPU

A modified generation script that timestamps multiple runs provides tokens‑per‑second data and RAM usage on the NPU.

Obtain a tokens‑per‑second figure and memory limit for the target model on the actual device

  1. Open Terminal and navigate to the folder containing the previous lesson’s generation script
  2. Edit the script to run generate() ten times with a fixed prompt and fixed output length, recording start and end timestamps for each run
  3. Execute the modified script several times and calculate tokens per second by dividing total generated tokens by total wall‑clock time
  4. Use htop (or an equivalent system monitor) to note RAM usage before and during generation, then record the largest model that fits within the NPU’s working set
  • You'll see A table showing each laptop’s tokens per second and peak RAM usage for the same prompt and model
  • Takeaway Tokens per second measured on your own hardware tells you whether the model can be deployed, not a generic TOPS spec
  • Check What modifications to the generation script allow you to calculate tokens‑per‑second and assess memory limits on the NPU?
  • Cost An afternoon per hardware tier, once — the timing script is a few lines, and after the first tier the process repeats quickly for the next.

2.6 Build a procurement case using existing NPU performance data

A short document quoting measured tokens‑per‑second and privacy benefits forms the basis of a hardware procurement case.

Create a budget request that starts with measured capability of the current fleet rather than a generic GPU ask

  1. Open PowerShell and run the performance script from lesson 5 to capture tokens‑per‑second on the current NPU fleet
  2. Copy the resulting figures into a short document that states: "Ryzen AI laptops generate X tokens/second on a 3B model locally with the fan off"
  3. Add a privacy note explaining that inference runs entirely on‑device, avoiding network calls and data residency concerns
  4. If a larger model is required, insert measured numbers for the target workload and compare them to the current fleet’s limits
  • You'll see A concise three‑sentence brief ready for a non‑technical decision maker
  • Takeaway Lead hardware requests with concrete numbers from devices you already own, only asking for new GPUs when a specific workload exceeds those limits
  • Check How do you use the performance figures from lesson 5 to craft a brief that justifies hardware requests based on existing NPU capability?
  • Cost Zero additional spend — the entire point is making the case with hardware already on the balance sheet.

2.7 Assess NPU inference limits

The 1–8B parameter INT4‑quantised model limit listed in vendor specifications defines the NPU’s inference ceiling.

Identify the practical ceiling of local NPU inference for a single user and model.

  1. Check model size against the hardware spec using 1–8B‑parameter, INT4‑quantized limits.
  2. Run a single‑request benchmark to confirm throughput for one user on the NPU.
  3. Verify model compatibility in the Lemonade catalog or via an OpenVINO IR conversion before deployment.
  • You'll see A clear statement that only small, supported models can run responsively on the laptop’s NPU.
  • Takeaway Small models, one user at a time, only the models the vendor toolchain actually supports, offline and at low power
  • Check Which criteria and benchmark steps determine the practical ceiling for local NPU inference for a single‑user model?
  • Cost Zero — this is the judgement call, and making it explicit costs nothing next to the cost of over-promising it later.

3See also

Big memory, slow bandwidth— the other non-default hardware path —Private AI for your org— the procurement conversation this chapter arms you for —Quantization as an engineering decision— NPUs are far pickier about format than a GPU is —

💬 Discuss this chapter

Ask, share, or report — over on the Heidelberg AI community forum.

Ask a question Stuck on something? The community can help. Show what you built Share it — others can try it and cheer it on. Report a problem or idea Spotted a bug, or have an improvement?