Heidelberg AICurriculum
Track 5 · Intermediate
5.5

Realtime voice agents

Talk to AI out loud — and have it talk back, live

8 lessons 2026-08-06 AI-generated

1Overview

How live speech-to-speech AI actually works: managed APIs you can call in minutes (OpenAI Realtime, Gemini Live, ElevenLabs), open-source pipelines you assemble yourself (Pipecat, Moshi), and the cascaded-vs-speech-native trade-off that decides whether your agent feels instant or merely fast.

A realtime voice agent is software you can actually talk to out loud and get a spoken answer back fast enough that it feels like a conversation, not a walkie-talkie exchange. You can rent this as a managed API (OpenAI, Google) or assemble it yourself from open pieces (Pipecat, Moshi) and run it on your own hardware. Under the hood, agents split into two families: cascaded pipelines chain separate speech-to-text, LLM, and text-to-speech stages — slower (~500ms) but easy to inspect and debug one stage at a time — while speech-native models like Moshi listen and speak with one model end-to-end, pushing latency under 200ms at the cost of being harder to customize. If privacy or cost is the priority, a fully local loop — whisper.cpp for transcription, a local LLM, Kokoro for speech — runs entirely on your own machine for $0 per minute. → Quick pick: fastest managed setup → OpenAI or Gemini Live; best voice → ElevenLabs; bring your own model APIs → Retell; open & self-orchestrated → Pipecat; lowest-latency speech-native → Moshi; fully private and local → whisper.cpp + a local LLM + Kokoro.

1.1After this chapter you can
→Explain the difference between cascaded (STT→LLM→TTS) and speech-native voice architectures
→Compare managed voice APIs (OpenAI Realtime, Gemini Live, ElevenLabs) on cost, latency, and quality
→Identify when a self-hosted open-source option (Pipecat, Moshi) beats a managed API
→Estimate realistic per-minute cost for a voice agent before building one
1.2What is a realtime voice agent?

It’s software you can speak to out loud and receive a spoken reply fast enough that the interaction feels like a natural conversation rather than a delayed back‑and‑forth.

1.3Which option provides the lowest latency?

The speech‑native Moshi model delivers the fastest response, pushing overall latency below two hundred milliseconds for a near‑instantaneous voice experience.

2Matrix 6 rows · 6 tools

openai-realtime
gemini-live
elevenlabs-convai
retell
pipecat
moshi
Runs locally / offline
no
no
no
no
yes
yes
Open source
no
no
no
no
yes
yes
Latency
150-400ms
100-200ms
~500ms
your stack
your stack
<200ms
Speech-native
yes
yes
no
no
no
yes
Cost per minute
~$0.25-0.35
~$0.07-0.12
$0.08 + LLM
$0.07 + stack
$0 + parts
$0 · GPU
Bring your own model
no
no
LLM only
yes
yes
self-host

3Lessons 8

3.1 Decide which voice‑agent architecture to use

Two shapes of voice agent: a cascaded pipeline that runs STT, an LLM and TTS in series and logs a transcript, and a speech‑native model that takes audio end‑to‑end and never shows one.

Select the appropriate voice‑agent architecture based on latency and inspectability needs

  1. Read the description of the cascaded pipeline and note its three stages: STT, LLM, TTS
  2. Read the description of the speech‑native model and note it processes audio end‑to‑end
  3. Compare the reported latencies (≈500 ms vs. <200 ms) and the presence or absence of a visible transcript
  • You'll see A stopwatch shows the speech‑native reply arriving noticeably sooner than the cascaded reply, while only the cascaded run displays a logged transcript
  • Takeaway Cascaded pipelines give ~500 ms latency with full text visibility, speech‑native models deliver under 200 ms but hide the transcript seam
  • Check What do you give up in exchange for the speech‑native model's shorter reply time?
  • Cost Free to reason about. The cost difference this creates is the subject of the rest of this chapter.

3.2 Identify what each managed voice API provides

OpenAI Realtime and Gemini Live stream audio in and out of one model; ElevenLabs Conversational AI wraps its own TTS around a separate transcription step and an external LLM.

Distinguish the capabilities of OpenAI Realtime, Gemini Live and ElevenLabs Conversational AI before comparing their costs

  1. Read the description of OpenAI Realtime API to see that it offers a persistent, bidirectional audio stream with no separate transcription step
  2. Examine the details of Gemini Live API to confirm it provides the same speech‑native streaming model at a lower price point
  3. Review the overview of ElevenLabs Conversational AI to understand its cascaded architecture that combines TTS, speech‑to‑text and an external LLM
  • You'll see A summary table that lists speech‑native streaming for OpenAI and Gemini versus a cascaded pipeline for ElevenLabs
  • Takeaway Speech‑native APIs stream audio directly while cascaded services wrap TTS around separate transcription and LLM components
  • Check Which of the three managed APIs is the cascaded one, and what does it wrap around its own voice layer?
  • Cost No infrastructure cost — you pay per minute or per token, covered next. The cost is being locked to a vendor's roadmap and pricing changes, which this chapter's data was current as of 2026-08-08 and will drift.

3.3 Compare managed voice API costs across providers

Voice APIs quote in different units — tokens for the speech‑native ones, minutes for the cascaded one — so nothing is comparable until both are converted to dollars per minute of audio.

Identify the per‑minute price of raw audio for Gemini Live, OpenAI Realtime and ElevenLabs and see why the cheapest option depends on your traffic pattern

  1. OPEN the Google Gemini pricing page in a browser and locate the rates for the Gemini 3.1 Flash Live Preview model
  2. READ the audio‑input price ($3.00 per 1M tokens or $0.005 /minute) and the audio‑output price ($12.00 per 1M tokens or $0.018 /minute)
  3. OPEN the OpenAI Realtime API pricing page, find the token rates for gpt‑realtime‑2.1 and gpt‑realtime‑2.1‑mini, then calculate a per‑minute range using 25 tokens / second
  4. OPEN the ElevenLabs pricing page, note the voice‑layer cost ($0.08–$0.10 /minute) and remember that LLM and telephony charges are separate
  • You'll see A table showing each provider’s quoted cost converted to dollars per minute of input and output audio
  • Takeaway Convert token‑based pricing to a common per‑minute basis before judging which service is cheaper
  • Check What must you convert every provider's quoted price to before the cheapest one is even visible?
  • Cost The numbers above, dated 2026-08-08. Treat any voice API price you have not personally re-checked against the vendor's current pricing page as stale — this space has repriced more than once a year.

3.4 Run a voice agent on Retell using your own language model

Retell is a hosted voice stack that leaves the reasoning step open: you pick the voice engine, the phone number and the language model separately.

Deploy a managed voice conversation that uses the LLM you choose for reasoning

  1. Open the Retell dashboard and click Create new agent
  2. Select the desired voice engine from the Voice Engine dropdown
  3. Choose the telephone number for inbound calls in the Phone Number dropdown
  4. Pick your preferred language model from the LLM dropdown
  5. Press Save configuration to launch the agent
  • You'll see A configuration screen showing separate dropdowns for voice engine, phone number and LLM
  • Takeaway Retell lets you combine any LLM with a hosted voice stack, so you get custom reasoning on a phone‑quality call at the cost of the voice service plus your model usage
  • Check Which three dropdowns does a new Retell agent need before you can save it?
  • Cost $0.07/minute for Retell's voice engine (verified 2026-08-08), plus your LLM's own per-token cost and telephony — realistic all-in figures from third-party trackers run $0.13–$0.31/minute.

3.5 Build a custom voice pipeline with Pipecat

Pipecat is a self‑hosted orchestrator: you instantiate a transport, an STT, an LLM and a TTS component and connect them as stages of one pipeline.

Create and run a self‑hosted Pipecat pipeline that connects your chosen STT, LLM and TTS services

  1. Install the Pipecat package with pip install pipecat
  2. Create a new Python script and import the required Pipecat classes
  3. Instantiate the transport layer you will use, for example DailyTransport()
  4. Add an STT component by creating an instance of your chosen provider’s SDK class
  5. Add an LLM component by instantiating the appropriate language model wrapper
  6. Add a TTS component using the vendor’s SDK class
  7. Connect all components in order with pipeline.add_stage() and start the pipeline with pipeline.run()
  • You'll see A Python file showing a Pipecat pipeline definition with stages for transport in, STT, LLM, TTS and transport out
  • Takeaway Pipecat lets you freely mix speech‑to‑text, language model and text‑to‑speech providers in one self‑hosted orchestration
  • Check Which stages does the pipeline need between transport in and transport out, and in what order?
  • Cost Pipecat itself is free and open source. Your actual cost is the sum of whichever STT/LLM/TTS vendors you plug in, plus the compute to run the orchestration process — there is no single per-minute Pipecat number.

3.6 Run Moshi locally for real‑time voice interaction

Moshi is a full‑duplex speech‑native model you run on your own GPU, so the audio never leaves the machine and there is no per‑minute fee.

Run Moshi on your own hardware to have a full‑duplex speech‑native conversation

  1. Open a terminal and git clone the repository kyutai-labs/moshi from GitHub
  2. Navigate into the cloned folder and pip install -r requirements.txt to set up dependencies
  3. Run the starter script with python run_moshi.py to launch a local Moshi session
  • You'll see Audio is captured from your microphone and played through your speakers while Moshi processes it locally with no network traffic visible
  • Takeaway Self‑hosting Moshi gives you full‑duplex, sub‑200 ms latency voice AI without per‑minute fees, but requires GPU time and operational overhead
  • Check What does self‑hosting Moshi buy you, and what does it cost you instead of a per‑minute fee?
  • Cost $0 per minute in vendor fees; the real cost is a GPU (and the ops work of keeping it running), not a line item on an invoice.

3.7 Calculate a realistic per‑minute cost for your voice agent

A realistic per‑minute figure adds three charges that are billed separately on a cascaded API — the voice engine, the LLM per exchange, and telephony — where a speech‑native API already includes the second.

Create a per‑minute cost estimate for your own voice agent using the chapter’s numbers

  1. Open Excel or Google Sheets
  2. Enter the formula (voice‑engine rate × minutes) + (LLM cost per exchange × exchanges per minute) + telephony cost in a cell
  3. Multiply that result by your expected call volume and compare it with data from a trial run
  • You'll see A spreadsheet showing four different total costs for the same conversation, each based on a different API and caching state
  • Takeaway Add voice‑engine, LLM and telephony costs separately for cascaded APIs; speech‑native APIs already include the LLM cost, and real‑world pauses inflate usage beyond scripted benchmarks
  • Check Which of the three cost lines disappears when you move from a cascaded API to a speech‑native one?
  • Cost No new cost — this is the arithmetic exercise that determines whether the vendor you are about to choose is affordable at your actual call volume.

3.8 Identify live‑voice failure modes

Three failures that only exist in live speech: barge‑in when the caller talks over the agent, turn detection firing at the wrong silence threshold, and network jitter breaking the audio up.

Spot barge‑in, turn detection errors and network jitter that only appear in spoken interactions

  1. Initiate a test call and let the caller speak over the agent mid‑sentence to observe barge‑in behaviour
  2. Adjust the silence‑threshold setting in your pipeline configuration to fine‑tune turn detection
  3. Simulate network jitter by throttling the connection during a test call and listen for choppy audio
  • You'll see A test call where the user interrupts the agent mid‑sentence and another call over a throttled connection, both sounding noticeably worse than the clean demo
  • Takeaway Barge‑in, turn detection and network jitter are live‑voice issues that never surface in text bots and must be tested with real interruptions and poor connections
  • Check Which two test conditions do you have to create deliberately, because a clean demo call never produces them?
  • Cost No direct cost — this is testing discipline. The cost of skipping it is a product that measures well in a demo and fails the first time a real caller interrupts it or has a bad connection.

4You’ll know it worked 8 checkable outcomes in this chapter

  • ✓Running the script in the terminal plays a greeting and responds to spoken prompts via console audio.
  • ✓Asking the agent for the current time or a specific search query triggers the decorated function and returns the result verbally.
  • ✓Asking the agent to search for real-world data triggers an MCP tool call and returns live results instead of mock data.
  • ✓Accessing the LiveKit cloud dashboard playground connects to your deployed agent and allows real-time voice interaction in the browser.
  • ✓The frontend connects, responds to voice input, and displays memory updates in the DOM.
  • ✓Asking follow-up questions or referencing earlier commands yields coherent responses without re-sending past messages.
  • ✓Speaking a trigger phrase causes the app to call an external API and return the result via audio and DOM update.
  • ✓The app correctly displays or speaks tool outputs even when they arrive after the initial assistant response.

5FAQ, Tips & How-to 8

one problem, one solution, one action
How-to Everyone

Need a local, customizable voice assistant

LiveKit abstracts real-time audio routing into a Python class, letting you chain STT, LLM, and TTS providers while keeping full control over conversation logic. By defining an entrypoint method and an AgentSession, you can manage room state, maintain history, and inject custom greetings without external orchestration. This architecture works because it decouples the voice pipeline from the agent's business logic, making it easy to swap providers or add custom behavior.

Cole Medin ↗ Lesson → AI-generated
How-to Everyone

Want your voice bot to tell the time

LiveKit treats standard Python functions as agent tools using a simple decorator, where the function's docstring acts as the prompt instructions for the LLM. This mirrors how other agent frameworks handle tool calling but integrates it directly into the voice pipeline without extra middleware. The LLM automatically matches user intent to the docstring and executes the function, returning results as spoken audio.

Cole Medin ↗ Lesson → AI-generated
How-to Everyone

Want a voice agent to pull real‑time data like Airbnb listings

LiveKit supports the Model Context Protocol (MCP) out of the box, allowing voice agents to dynamically discover and call external tools without writing custom Python wrappers. By pointing the agent to an MCP gateway URL, LiveKit handles tool discovery, parameter validation, and execution routing. This works because MCP standardizes how LLMs interact with external data sources, making it trivial to plug in live APIs like search engines or databases.

Cole Medin ↗ Lesson → AI-generated
How-to Everyone

Need a voice bot you can speak to from a web page or phone

LiveKit provides a CLI that handles containerization, environment variable management, and cloud deployment, turning a local Python script into a production-ready web-accessible agent. The CLI abstracts away Docker and infrastructure setup by generating a Dockerfile, uploading secrets securely, and provisioning a scalable cloud instance. This works because it follows a standard app lifecycle (env -> start -> create) that mirrors modern serverless deployment patterns.

Cole Medin ↗ Lesson → AI-generated
How-to Everyone

Need a hands‑free voice assistant on your PC

The video demonstrates cloning a starter repository, installing dependencies, and configuring a .env file with an OpenAI API key. A relay server bridges the frontend WebSocket connection to OpenAI’s backend, enabling real-time two-way audio without manual button presses.

Developers Digest ↗ Lesson → AI-generated
How-to Everyone

Unlike traditional chat APIs that require appending the entire message history to each request, the Realtime API maintains server-side state. This allows natural follow-ups and memory commands without manually managing context windows or token limits.

Developers Digest ↗ Lesson → AI-generated
How-to Everyone

Want your voice bot to run custom actions

You can extend the Realtime API by defining custom tools using natural language descriptions and JSON schemas. The video shows how to append these tools to the WebSocket connection, allowing the model to trigger external APIs and stream results back into the conversation.

Developers Digest ↗ Lesson → AI-generated
How-to Everyone

Because the Realtime API streams audio and processes function calls asynchronously over WebSockets, tool invocations may arrive after the model has already started responding. Developers should design UIs and logic to gracefully handle or wait for tool outputs before finalizing responses.

Developers Digest ↗ Lesson → AI-generated

The same set on /recipes, filtered by tool and role.

6Videos 2

7FAQ 24

How do I run a basic voice AI agent on my computer?

Import the LiveKit dependencies, create an Assistant class that inherits from livekit.agents.Agent, and set a system prompt in its initializer. Define an @entrypoint method that builds an AgentSession with your chosen speech‑to‑text (STT), language model (LLM), and text‑to‑speech (TTS) providers, then start the session and generate an initial greeting. Finally, launch the script with livekit.cli.run_app(entrypoint).

Can I give my voice agent custom abilities like checking the time?

Yes. Write a regular Python function inside your agent class that performs the desired action, add the @livekit.agents.function_tool decorator above it, and provide a detailed docstring describing its purpose and parameters. LiveKit uses the docstring as a prompt for the LLM, which will call the function when the user’s request matches the description, and the result is spoken back to the user.

What is needed to connect my voice agent to external APIs without writing wrappers?

LiveKit supports the Model Context Protocol (MCP), which lets you point your agent to an MCP gateway URL. LiveKit then discovers the available tools, validates parameters, and routes execution automatically during AgentSession setup. This enables live queries to services such as search engines or databases with minimal code.

How can I make my voice agent accessible from a web browser?

Install the LiveKit CLI, log in with lk cloud login, and upload your environment variables using lk app env. Run lk start to generate the deployment manifest, then execute lk agent create which builds a Docker container and deploys it to LiveKit’s cloud. Once deployed, you can test the agent through the browser playground or a phone number.

What advantage does the OpenAI Realtime API have for conversation memory?

The Realtime API keeps the conversation state on the server, so you don’t need to resend the full message history with each request. This lets the model remember prior turns and respond to memory commands like “set memory to buy eggs” naturally, avoiding token‑limit concerns.

How do I add my own function calls to a Realtime API voice assistant?

Define a tool schema that includes a natural‑language description and a JSON schema for required parameters. Append this tool to the WebSocket connection with the add tools method, then implement a handler that executes the external call (e.g., a weather API) and returns the payload to the model. The model can incorporate the result into its spoken response.

What should I do if a tool’s response arrives after the model has started speaking?

Monitor WebSocket messages for tool‑call requests and buffer any partial audio until the tool’s data is received. Once the payload arrives, update the audio stream or UI with the completed information, and consider retry or fallback logic for delayed responses.

How do I start a basic voice AI that listens and talks back in real time?

Create a Python class that inherits from livekit.agents.Agent, set a system prompt, and define an @entrypoint method that builds an AgentSession with your chosen speech‑to‑text (STT), language model (LLM), and text‑to‑speech (TTS) providers. Call session.start() and then use session.generate_reply() to send an initial greeting. Run the script with livekit.cli.run_app(entrypoint).

Can I add custom Python functions that my voice agent can call, like checking the time?

Yes. Write a normal Python function inside your agent class, add the @livekit.agents.function_tool decorator, and provide a detailed docstring describing what the function does and its parameters. LiveKit uses the docstring as instructions for the LLM, which will match user requests to the function and execute it, returning the result as spoken audio.

What if I want my voice agent to use live external data, such as real‑time listings?

LiveKit supports the Model Context Protocol (MCP), which lets your agent discover and call external tools without writing extra wrappers. Point the AgentSession to an MCP gateway URL (set up via Docker or another method) and include any required authentication; LiveKit will register those tools automatically, allowing you to ask questions that trigger live data retrieval.

How can I deploy my voice agent so anyone can use it from a web browser?

Install the LiveKit CLI, log in with lk cloud login, and upload your environment variables (API keys, model settings) using lk app env. Then run lk start to generate the deployment manifest and lk agent create to build and push a Docker container to LiveKit’s cloud. The deployed agent can be accessed through the cloud dashboard’s playground or via a phone number.

How can I chain speech‑to‑text, language model, and text‑to‑speech in a LiveKit voice agent?

LiveKit provides an Agent class where you define an @entrypoint method that creates an AgentSession with the STT, LLM, and TTS providers you choose. The session.start() call opens the audio room, and generate_reply() triggers the pipeline so spoken input is transcribed, processed by the model, and spoken back in real time.

How do I give my voice agent custom capabilities using Python functions?

Define a regular Python function inside your agent class and decorate it with @livekit.agents.function_tool. The function’s docstring describes its purpose and parameters; the LLM reads this description and calls the function when user intent matches, then LiveKit returns the result as spoken audio.

What is MCP integration and why would I use it with a LiveKit voice agent?

MCP (Model Context Protocol) lets a LiveKit agent discover external tools automatically by pointing to an MCP gateway URL. LiveKit handles tool discovery, validates parameters, and routes execution, so the agent can query live APIs—like real‑time listings—without writing custom wrappers.

How can I ship my LiveKit voice agent so users can talk to it in a web browser?

Install the LiveKit CLI, log in, upload your environment variables with lk app env, then run lk start followed by lk agent create. The CLI builds a Docker container, provisions a cloud instance, and makes the agent reachable through a browser‑based playground or phone number.

How does the OpenAI Realtime API keep conversation context without sending the full history each turn?

The Realtime API maintains server‑side state for the session, so after you connect via WebSocket you can speak naturally and the model remembers previous turns. Memory commands (e.g., “set memory to buy eggs”) are stored on the server, allowing accurate follow‑ups without resending the entire chat log.

How do I start a basic voice AI that listens and talks back in real time?

Create an Assistant class that inherits from livekit.agents.Agent, set a system prompt in the constructor, and define an @entrypoint method that builds an AgentSession with your chosen speech‑to‑text (STT), language model (LLM), and text‑to‑speech (TTS) providers. Call session.start() and then session.generate_reply() to send an initial greeting. Run the script with livekit.cli.run_app(entrypoint).

Can I add custom Python functions that my voice agent can call?

Yes, write a normal Python function inside your agent class, add the @livekit.agents.function_tool decorator, and provide a detailed docstring describing what the function does and its parameters. LiveKit uses the docstring as instructions for the LLM, which will match user requests to the function and execute it, returning the result as spoken audio.

What is needed to let my voice agent fetch live data from external APIs?

Set up a Model Context Protocol (MCP) server that exposes tools over HTTP, then include its URL and any authentication info in the AgentSession configuration. LiveKit will automatically discover those MCP tools, validate parameters, and route calls, so you can ask questions that trigger live API queries without writing extra wrappers.

How can I deploy my voice agent so anyone can use it from a browser?

Install the LiveKit CLI, log in with lk cloud login, upload your environment variables using lk app env, and run lk start to generate the deployment manifest. Then execute lk agent create, which builds a Docker container and pushes it to LiveKit’s cloud. After deployment you can open the web playground or use a phone number to interact with the agent via microphone.

+ 4 more in the library.

8Glossary 72 terms

Show the 72 terms
OpenAI Realtime API
LiveKit
A software platform that handles real‑time audio routing and lets you connect speech‑to‑text, language models, and text‑to‑speech services.
AgentSession
An object that represents a single voice conversation session and coordinates the chosen speech and AI components.
@entrypoint
A Python decorator that marks the method where the voice agent starts running.
livekit.cli.run_app(entrypoint)
A command‑line call that launches your agent script using the function you marked with @entrypoint.
@livekit.agents.function_tool
A decorator that tells LiveKit to treat a regular Python function as a tool the language model can call during a voice chat.
docstring
The text placed right under a function definition that describes what the function does and how to use it.
MCP (Model Context Protocol)
A standard way for language models to discover and call external APIs without custom code.
.env file
A plain‑text file that stores configuration values such as API keys, which the program reads at startup.
relay server
A small backend service that forwards audio data between your web app and OpenAI’s Realtime API over WebSocket.
WebSocket
A persistent internet connection that lets the browser send and receive audio data instantly without reloading the page.
tool schema
A JSON description that tells the model what a custom function does, what inputs it needs, and how to call it.
add tools method
A LiveKit command used on an open WebSocket connection to register your custom tool schemas so the model can invoke them.
Google Gemini Live
LiveKit
A Python library that handles real‑time audio routing and lets you connect speech‑to‑text, language models, and text‑to‑speech services.
AgentSession
An object that represents a single voice conversation session and manages the chosen audio pipeline providers.
@entrypoint
A decorator that marks the method LiveKit should call first when starting an agent.
livekit.agents.Agent
The base class you inherit from to create your own voice assistant logic.
livekit.cli.run_app(entrypoint)
A command‑line call that launches the agent using the method marked with @entrypoint.
@livekit.agents.function_tool
A decorator that tells LiveKit to expose a regular Python function as a tool the language model can call.
docstring
The text placed right under a function definition that describes what the function does and its parameters; LiveKit uses it as instructions for the model.
MCP (Model Context Protocol)
A standard way for language models to discover, validate, and call external tools over HTTP.
.env
A plain‑text file that stores environment variables such as API keys so they are not hard‑coded in the script.
Dockerfile
A text file that tells Docker how to build a container image for your agent, including dependencies and start commands.
lk cloud login / lk app env / lk start / lk agent create
LiveKit CLI commands that log you into the cloud service, upload secret variables, prepare deployment settings, and build & launch a containerized agent.
WebSocket
A persistent network connection that lets the frontend send audio to the server and receive responses instantly without re‑loading the page.
ElevenLabs Conversational AI
LiveKit
A software platform that lets you build real‑time voice applications by handling audio routing and connections for you.
AgentSession
An object that represents a single conversation instance, keeping track of the room state and history while running the voice pipeline.
@entrypoint
A decorator that marks a method as the starting function that LiveKit will call when launching your agent.
livekit.agents.Agent
The base class you inherit from to create your own voice assistant, providing built‑in hooks for speech‑to‑text, language model, and text‑to‑speech integration.
@livekit.agents.function_tool
A decorator that tells LiveKit to treat the following Python function as a tool the assistant can call when the user asks for it.
docstring
The text placed right under a function’s definition that describes what the function does; LiveKit uses this description to match user requests to tools.
MCP
Short for Model Context Protocol, a standard way for language models to discover and call external APIs without custom code.
.env file
A simple text file where you store configuration values like API keys so the program can read them securely at runtime.
WebSocket
A network connection that stays open, allowing real‑time two‑way communication between your app and a server such as OpenAI’s Realtime API.
tool schema
A JSON description that defines what a custom function does, its required inputs, and how the model should call it.
lk cloud login
A command in the LiveKit CLI that authenticates you with your LiveKit cloud account so you can deploy apps.
add tools method
A function call on the WebSocket connection that registers your custom tool schemas so the Realtime model can invoke them during a conversation.
Retell AI
LiveKit
A platform that provides Python classes and tools for building real‑time voice applications.
AgentSession
An object that holds the state of a voice conversation, including history and active audio pipelines.
@entrypoint
A decorator that marks a method as the starting function for launching an agent session.
livekit.agents.Agent
The base class you inherit from to create your own voice assistant logic.
livekit.cli.run_app(entrypoint)
A command‑line call that runs the specified entrypoint method to start the agent.
@livekit.agents.function_tool
A decorator that tells LiveKit to treat a regular Python function as a callable tool for the assistant.
docstring
The text placed right under a function definition that describes what the function does and its parameters.
MCP (Model Context Protocol)
A standard way for language models to discover, validate, and call external tools over HTTP.
.env
A file that stores environment variables such as API keys in a simple key‑value format.
Dockerfile
A script that tells Docker how to build a container image for your application.
WebSocket
A persistent internet connection that lets the client and server exchange audio or data instantly in both directions.
relay server
A small backend program that forwards WebSocket messages between your front‑end app and an external API like OpenAI’s Realtime service.
Pipecat
livekit.agents.Agent
A base Python class provided by LiveKit that you extend to create a custom voice AI agent.
@entrypoint
A decorator that marks the method where the agent’s conversation session is started.
AgentSession
An object that manages a single voice interaction, handling audio routing and tool calls for the agent.
livekit.cli.run_app(entrypoint)
A command‑line helper that launches your Python script by calling the function you marked with @entrypoint.
@livekit.agents.function_tool
A decorator that tells LiveKit to treat a regular Python function as a tool the LLM can call during conversation.
MCP
Short for Model Context Protocol, a standard that lets the agent discover and invoke external APIs without custom code.
Dockerfile
A text file generated by the LiveKit CLI that describes how to build a container image for your agent.
.env
A simple configuration file where you store secret keys and settings as name‑value pairs.
OpenAI Realtime API
An OpenAI service that streams audio to and from a model over a WebSocket, enabling live voice interactions.
WebSocket
A network connection that stays open so the client and server can exchange messages instantly in both directions.
JSON schema
A structured description of a tool’s input parameters written in JSON format, used by the Realtime API to validate calls.
add tools method
A function you call on the WebSocket connection to register custom tool definitions so the model can invoke them.
Moshi (Kyutai)
LiveKit
A software platform that provides building blocks for real‑time audio and video applications.
livekit.agents.Agent
A base Python class you extend to create a voice AI agent with LiveKit.
AgentSession
An object that manages the state of a conversation, including audio pipelines and history, for one user session.
@entrypoint
A decorator that marks the method LiveKit should call first when starting an agent.
livekit.cli.run_app(entrypoint)
A command‑line function that launches your script by calling the method you marked with @entrypoint.
@livekit.agents.function_tool
A decorator that tells LiveKit to expose a regular Python function as a tool the LLM can call.
docstring
The text placed right under a function definition that describes what the function does and its parameters.
MCP
Short for Model Context Protocol, a standard way for language models to discover and call external tools.
Dockerfile
A text file that tells Docker how to build a container image for your application.
.env
A file that stores environment variables such as API keys, which the program reads at startup.
WebSocket
A network connection that stays open so client and server can exchange messages instantly in both directions.
add tools method
A LiveKit function you call on the WebSocket to register custom tool definitions so the model can invoke them during a voice session.

9See also

Voice: text ↔ speech— the TTS+STT building blocks —Run AI privately— run the whole loop locally —Agent frameworks— give the voice agent tools —

💬 Discuss this chapter

Ask, share, or report — over on the Heidelberg AI community forum.

Ask a question Stuck on something? The community can help. Show what you built Share it — others can try it and cheer it on. Report a problem or idea Spotted a bug, or have an improvement?