runtime
Text Generation Inference
Text Generation Inference (TGI) is an open‑source toolkit that lets you deploy and serve popular open‑source large language models such as Llama, Falcon, BLOOM or GPT‑NeoX. It runs on Nvidia, AMD, Intel GPUs, AWS Trainium/Inferentia, Google TPUs or CPUs via Docker, and provides features like tensor parallelism, token streaming (SSE), continuous batching and quantization. The software is free to use; you only need the hardware and any cloud‑provider costs for the underlying compute.
What it can help with
- inference server
- continuous batching
- token streaming
- tensor parallelism
- hugging face