runtime · Ettore Di Giacinto + the LocalAI community
LocalAI
LocalAI is the front door: one small core binary that speaks the OpenAI and Anthropic APIs, and routes each request to the right backend behind the scenes — llama.cpp for one model, vLLM for another, whisper.cpp for audio, Stable Diffusion for images — all through the same endpoint. Backends ship as OCI images and run as isolated gRPC processes, so a crash in one never takes down the others, and you install/upgrade a backend without touching the core. That's the concrete team win: n8n, AnythingLLM, or your own scripts all point at one stable URL forever, even as you swap which engine actually answers underneath — a new GPU, a new quantization, a different model family, none of it requires reconfiguring every downstream tool. It runs on NVIDIA, AMD, Intel, or Vulkan GPUs with autodetection, or CPU-only, and as of 2026 adds WebRTC real-time audio, MCP-based agents, and speaker diarization.
What it can help with
- api gateway
- model routing
- gpu inference
- cpu fallback
- audio transcription
- image generation
- backend isolation
- real-time audio
- quantized models