Heidelberg AI Curriculum

16tools in the catalogue

AI Bench availability is not confirmed for these catalogue entries.

vLLM

Runtimes · Self-hosted

FreeOpen source

Good for

Serve language models to many simultaneous users from shared GPU hardware.
Use paged attention, batching, caching, and quantization to manage inference.
Expose OpenAI-compatible, Anthropic, or gRPC interfaces for applications.

Watch out

It is GPU-first and operationally oriented toward concurrent service workloads.

Official site

Text Generation Inference

Runtimes · Self-hosted

Free

Good for

Deploy open language models behind an inference service using Docker.
Stream tokens and serve concurrent requests with batching and parallelism.
Use quantization across supported GPUs, accelerators, or CPUs.

Watch out

TGI is in maintenance mode, so expect limited new feature development.

Official site

Podman

Runtimes · Desktop · Self-hosted

FreeOpen source

Good for

Run OCI containers without a central daemon.
Use rootless containers when your workflow benefits from reduced privileges.
Keep a familiar Docker-compatible command-line workflow for images and pods.

Watch out

podman compose depends on an external provider whose behavior varies by environment.

Official site

OpenVINO GenAI

Runtimes · Self-hosted

Free

Good for

Run language, vision, image, speech-to-text, and text-to-speech models locally.
Use Python, C++, or Node.js on CPUs, Intel GPUs, or NPUs.
Apply OpenVINO optimizations such as speculative decoding and KV-cache eviction.

Watch out

Choose compatible models and hardware before building an OpenVINO workflow.

Official site

Open WebUI

Runtimes · Self-hosted

Free tier

Good for

Give teams a self-hosted chat interface for local models.
Control model visibility with roles and user groups.
Connect Ollama, LocalAI, vLLM, LM Studio, or compatible APIs.

Watch out

Self-hosting and enterprise branding requirements vary; check the current licence terms.

Official site

ONNX Runtime GenAI

Runtimes · Self-hosted

Free

Good for

Run generative models on devices through ONNX Runtime.
Integrate with bindings across several programming languages.
Target CPUs, GPUs, NPUs, and web accelerators across platforms.

Watch out

Some features require building from source rather than installing a prebuilt package.

Official site

Ollama

Runtimes · Self-hosted

FreeOpen source

Good for

Run open language models locally from the command line.
Expose a localhost API for scripts, apps, and lab services.
Use GGUF models for offline analysis and drafting tasks.

Watch out

Ollama is CLI- and API-first; add a separate interface for graphical chat.

Official site

NVIDIA System Management Interface

Runtimes · Self-hosted

Free

Good for

Inspect identity and telemetry for supported NVIDIA GPUs.
Check utilization, memory use, and temperature during workloads.
Record commands and intervals as reproducible capacity evidence.

Watch out

A single final sample misses peak memory; sample throughout runs when capacity matters.

Official site

LocalAI

Runtimes · Self-hosted

FreeOpen source

Good for

Route OpenAI and Anthropic API requests to local backends.
Keep downstream clients on one stable endpoint.
Isolate backend processes for models, audio, or images.

Watch out

It does not improve backend throughput; performance still depends on the selected engine.

Official site

LM Studio

Runtimes · Self-hosted

Free

Good for

Download and run compatible models through a desktop interface.
Expose a local OpenAI-compatible API for applications.
Use headless mode when a graphical session is unnecessary.

Watch out

Local execution depends on configuring models, extensions, updates, and integrations to stay local.

Official site

llm-d

Runtimes · Self-hosted

Free

Good for

Serve language models across accelerators with Kubernetes.
Deploy vLLM or SGLang using ready-made Helm charts.
Turn single-node servers into distributed serving clusters.

Watch out

You need a Kubernetes cluster and the operational skills to run distributed services.

Official site

llama.cpp

Runtimes · Self-hosted

FreeOpen source

Good for

Run GGUF models through a configurable local inference engine.
Tune builds, quantization, and hardware-specific acceleration.
Expose chat, completion, and embedding endpoints through its server.

Watch out

It has no built-in account or UI; compiling and tuning require hands-on setup.

Official site

llama-swap

Runtimes · Self-hosted

FreeOpen source

Good for

Switch model backends behind one compatible proxy endpoint.
Configure model startup and unloading with YAML commands.
Share one GPU across models without keeping every process loaded.

Watch out

A model swap introduces cold-start time, so it suits variable rather than instant workloads.

Official site

Jan

Runtimes · Self-hosted

FreeOpen source

Good for

Chat with locally running models through an offline-first desktop app.
Download models from its built-in Hub and keep work on your machine.
Add a cloud key per thread when local inference is insufficient.

Watch out

Local models need capable hardware, and its curated Hub is comparatively smaller.

Official site

Docker

Runtimes · Desktop · Self-hosted

Free tier

Good for

Build, run, and share OCI container workloads from the command line.
Choose it when applications need repeatable packaged environments.
Compose and registries support multi-container development and image distribution.

Watch out

Docker products have separate terms, and daemon or socket access is privileged.

Official site

Caddy

Runtimes · Self-hosted

FreeOpen source

Good for

Route web traffic through a reverse proxy that can obtain HTTPS automatically.
Choose it when one TLS entry point should front private application ports.
Its automatic certificates reduce routine certificate configuration work.

Watch out

Automatic HTTPS needs correct DNS, reachable challenges, and a reviewed network policy.

Official site