ONNX Runtime GenAI is Microsoft's free library for running generative models on-device, handling tokenization, sampling, KV-cache and the generation loop around ONNX-format models. Pick it when you're embedding generation directly into a Python, C# or C++ application and need token-level control, not a ready-made chat app. It powers Microsoft's own Foundry Local, Windows ML and VS Code AI Toolkit, but its API is still preview and model artifacts are hardware-specific.
Other good fits
Building a streaming UI that needs to decide when to stop generating token by token.
+ Pros
Read pros
ONNX Runtime GenAI manages the full generation workflow around ONNX-format models: preprocessing, inference, token search and sampling, KV-cache management, and grammar-based decoding for structured output. It provides APIs in Python, C# and C/C++ (with Java needing a source build and Objective-C still in development), giving you token-level control that's useful for a streaming interface or an application that needs to decide precisely when to stop generating. It's the library behind several of Microsoft's own products, including Foundry Local, Windows ML and the VS Code AI Toolkit.
Pick it when…
You're building an application in Python, C# or C++ and need generation embedded directly in your process, not called over HTTP.
You need fine control over the generation loop: stopping conditions, streaming output, or grammar-constrained (structured) output.
You're comfortable preparing ONNX model artifacts specifically for your target hardware.
− Cons
Read cons
The runtime doesn't repair a model's incorrect answers; a successful sample run only proves the pipeline works, not that outputs are good. Longer-context model variants consume more memory, and your application code, not the library, is responsible for limits, interruption handling and rendering partial output sensibly. Model artifact preparation is genuinely execution-provider-specific work, not a one-time setup you can skip by downloading one generic file.
Pick something else when…
You want a simpler HTTP-service abstraction instead of direct token-loop control.Ollama gives you that when you don't need in-process embedding.
You're working with GGUF-format models specifically.llama.cpp is the more natural fit for that ecosystem.
You want another on-device library with a different model-preparation workflow to compare. OpenVINO GenAI offers an alternative approach worth evaluating against ONNX Runtime GenAI's.
Avoid if
Your project can't tolerate preview API changes or the extra work of matching model artifacts to your exact hardware.
Requirements
Models must be converted to ONNX format and typically quantized before use; you download prepared artifacts (for example, a quantized Phi-3 model) from a source like Hugging Face rather than pointing the runtime at arbitrary weights. Artifacts are hardware- and execution-provider-specific: a CUDA-exported model isn't interchangeable with a CPU or DirectML setup, so match your package, example version and model artifact together, since the project's own README warns that main-branch examples may not align with the latest stable release (currently v0.11.5).
First run
Documentation-based suggestion — not run here.
Run the official Phi-3 sample before building your own integration.
Install the correct ONNX Runtime GenAI package for your target (CPU, CUDA or DirectML) and download the matching Phi-3 ONNX artifact.
Use the official phi3-qa.py sample from a release tag matching your installed package version.
Run it with a short, clearly fictional prompt.
Watch the streamed output and check it stays within the premise you gave it.
Done when you've seen one complete streamed response from a matched package-and-artifact pair, confirming your environment is correctly set up before you write your own integration code around it.
Integration
Use the Python, C# or C/C++ API directly in your application code; the generator classes let you append tokens and pull output incrementally rather than waiting for a complete response. The API is explicitly marked preview and subject to change, so pin a specific version rather than tracking the latest release automatically in production code.
Check the result
A successful sample run confirms that configuration can generate text; it says nothing about output quality, speed on your actual hardware, or device coverage beyond what you tested. If the model or API fails to load, first check that your artifact, example version and installed package all match, rather than installing several execution-provider packages at once hoping one works.
Links & evidence
Documentation-based · not hands-on · Sep 23, 2026
Rests on the ONNX Runtime GenAI GitHub repository, checked 23 September 2026. No package was installed and no model was run for this review.
ONNX Runtime GenAI repository · Sep 23, 2026 — Generation-loop scope, current stable version (v0.11.5), supported languages, model preparation, real-world usage (Foundry Local, Windows ML, VS Code AI Toolkit).
ONNX Runtime GenAI documentation · Sep 23, 2026 — Preview API status, generate() method, token-by-token control, sampling strategies, tutorials and language bindings.
ONNX Runtime GenAI Python API · Sep 23, 2026 — Model and tokenizer classes, generator parameters, incremental token generation.
Further review notes & alternatives
Costs and operating effort
The library is free. Local inference costs are your device's compute, electricity and storage; there's no hosted-inference charge for the local path, though rented hardware (if you use it) has its own cost. The repository's own telemetry notice states it may collect usage data and send it to Microsoft, which is a separate question from whether your prompts stay local; check the privacy documentation if that distinction matters for your project.
Alternatives and trade-offs
Ollama wins when an HTTP-service abstraction is enough and you don't need in-process, token-level control.
llama.cpp wins when your models are in GGUF format rather than ONNX.
LM Studio wins when you want a ready-made desktop chat interface instead of a library to build one with.