NVIDIA System Management Interface runtime decision matrix
Overview
nvidia-smi is NVIDIA's free, driver-installed command-line tool for reading GPU state: memory used, utilization, temperature, power draw and running processes. Pick it as the first diagnostic step before assuming a slow or failing local model is the model's fault. It's NVIDIA-only, its table output isn't a stable API to script against, and one snapshot doesn't prove peak usage.
Other good fits
A quick, no-install diagnostic snapshot on any machine with an NVIDIA driver already installed.
+ Pros
Read pros
nvidia-smi (NVIDIA System Management Interface) reads GPU state through the installed driver: frame-buffer memory (total, used, available), GPU and memory utilization percentages, power draw, temperature, clock frequencies, and which processes are currently using the GPU. It ships with every supported NVIDIA driver at no extra cost, and with appropriate privileges can also change some device settings, though read-only inspection is what most people need first.
Pick it when…
A local model is running slowly or failing to load, and you want to check GPU memory and utilization before assuming the model itself is broken.
You want a quick readiness check on a machine before starting a GPU-dependent task.
You need a no-install diagnostic that just works on any machine with a working NVIDIA driver.
− Cons
Read cons
Supported fields vary by device and driver version, and NVIDIA explicitly allows unavailable fields to show N/A rather than guaranteeing every field on every card. The tool tells you nothing about whether a model's output is correct, or whether a slowdown is actually CPU-bound rather than GPU-bound. Don't reset a GPU or change its power settings just to make a diagnostic check pass, especially on a shared machine where that could disrupt someone else's work.
Pick something else when…
You need a durable, cross-driver programmatic interface. Use NVML directly instead of scraping nvidia-smi's text output, which can change format between driver releases.
You just need a quick visual check and don't have NVIDIA hardware. Your operating system's own task or activity monitor covers general resource use.
The problem might be in a container, not the host. Check your container runtime's own GPU-passthrough diagnostics alongside nvidia-smi.
Avoid if
You're building software that scripts against the CLI's text output long-term; that format isn't guaranteed stable across driver versions.
Requirements
nvidia-smi needs a supported NVIDIA GPU with a working driver already installed; it doesn't work on other GPU vendors' hardware. No model, dataset or cloud account is required to run it, just a terminal on a machine you're authorized to inspect.
First run
Documentation-based suggestion — not run here.
Read first run
Write a short GPU-readiness note before starting a local inference task.
Run nvidia-smi for a summary view: driver version, GPU name, memory used/total, and any running processes.
Run nvidia-smi -q -d MEMORY,UTILIZATION for a more detailed memory and utilization breakdown.
Note the device name, memory currently in use, and whether utilization was high at that moment.
Decide whether you actually have enough free memory for the model you're about to run.
Done when you have a short written note (device, memory free, utilization) that tells you whether it's worth attempting the model load, rather than just trying and guessing why it failed.
Integration
For a one-off human check, run the CLI directly. For software that needs to keep working across driver upgrades, use NVML (NVIDIA Management Library) or its Python bindings instead of parsing nvidia-smi's text table: NVIDIA explicitly states the CLI's output format isn't guaranteed backward-compatible, while NVML's API is maintained for that purpose.
Check the result
Compare the free memory shown against what your target model needs. A field showing N/A isn't necessarily a failure; some fields aren't supported on every device or driver combination. A single snapshot can miss a short burst of activity, so if timing matters, use the -l loop flag to sample repeatedly rather than trusting one read.
Links & evidence
Documentation-based · not hands-on · Sep 23, 2026
Rests on NVIDIA's SMI overview, command reference and NVML API documentation, checked 23 September 2026. No GPU query or state change was performed for this review.
NVIDIA SMI overview · Sep 23, 2026 — Driver-distributed CLI for monitoring and privileged management, built on NVML.
NVML API reference · Sep 23, 2026 — Documented query, management and event API families for programmatic monitoring.
Further review notes & alternatives
Costs and operating effort
The tool itself is free and ships with the driver; there's no separate installation or license. The only cost is the machine's normal electricity or rental cost, which you're already paying. Effort is mostly keeping your GPU driver current and correctly interpreting what you see: a high memory figure can mean allocated buffers rather than active computation, and low utilization alone doesn't tell you where a bottleneck actually is.
Alternatives and trade-offs
NVML (and its language bindings) wins when you're building monitoring software that needs to survive driver upgrades without breaking.
An operating-system task monitor wins for a fast, general visual check when you don't need GPU-specific detail.
Podman or Docker documentation on GPU passthrough is the next place to look if nvidia-smi looks fine on the host but a containerized workload still can't see the GPU.