What it takes to run an AI agent for real — not just demo it
The gap between an agent that works in a demo and one you can trust in production — plus the "agent as a readable folder" pattern (markdown instructions + skills + a few typed tools) that frameworks like Vercel eve use to close it.
A demo agent is a prompt and a loop; a production agent is one you can trust to run unattended — on a schedule, over Slack, touching real data. This chapter is the demo→production gap: durability (it survives a crash mid-task and resumes), evaluations (you catch regressions before users do), approvals (a human signs off before anything expensive or destructive), a sandbox, and a one-command deploy. The unifying idea — shown with Vercel's open-source eve as the running example — is that a whole agent is now a folder you can read: plain-markdown instructions and skills plus a few typed tools, with the framework handling the hard parts. → Unlike "Agent frameworks" (which builder to pick) or "Skills, tools & extensions" (the shared building blocks), this chapter is what turns any of them from a demo into something production-grade. Tool-agnostic; eve is just the clearest current example.
The moment an agent leaves the chat window to run for real, a prompt-and-a-loop is no longer enough.
Durability, evaluations, approvals, a sandbox, channels/schedules, and one-command deploy — six pieces, each expressed as plain markdown or a typed tool the framework wires together.
Defining an agent is becoming less about plumbing and more about writing clear instructions and curating skills — both just markdown — while the framework handles the production hard parts.
A simple key‑value store saved as JSON in the agent folder that survives restarts.
You will be able to read and write durable data from any tool in your Eve agent.
agent/state.json inside your existing Eve project and add {} as its initial content.fs/promises module (built‑in) by adding import { readFile, writeFile } from "fs/promises"; at the top of a new tool file agent/tools/save_state.ts.defineTool that accepts a key: string and value: any, reads state.json, updates the object, and writes it back.load_state.ts that takes a key: string, reads state.json, and returns the stored value (or null).npm run dev, invoke each tool from the chat UI, and verify that values persist after stopping and restarting the process.Agent Span is a server that stores the execution state of an AI agent so it survives crashes and can be resumed from a web UI.
Start an Agent Span server, run your agent against it, and resume a stopped execution through the UI
The Evaluation Simulator runs predefined user scenarios against an agent and records inputs, outputs, and tool calls to build a golden dataset for quality tracking.
Produce a golden dataset of agent interactions and detect regressions before deployment
An end‑to‑end (e2e) test that runs the agent against predefined inputs and checks its outputs.
You will be able to run automated regression tests that catch unexpected changes in your agent’s behavior.
e2e and add a file weather.test.ts.import { runAgent } from "eve"; and write a test that sends the prompt “What is the weather in Paris?” to the agent.Sunny (the mock weather condition defined in the example tool).package.json: `LangGraph’s deploy CLI packages an agent, builds a container, provisions a hosted API on LangSmith, and enables tracing and versioning automatically.
Ship the evaluated agent to a scalable production endpoint using a single CLI command
78 outcomes in all — one per recipe below.
Agent dies on script crash
Agent Span moves the execution state of an AI agent from your local process to its own server, so the workflow persists even if the script or server restarts. This gives you full visibility into each step and lets you pause, resume, or inspect runs via a web UI.
Ollama EveryoneNeed a local LLM for your AI agents
Ollama provides an easy way to run large language models locally, eliminating external API latency and cost. By pulling Llama 3.1 you get a capable open‑source model that Agent Span can call directly.
Need a simple web form for AI terms
A minimal Flask front‑end lets users submit terms, forwards the request to the Agent Span‑managed agent, and displays structured explanations. Because the backend agent runs on the resilient server, the UI remains responsive even if the Python process restarts.
Documents may have malware, PII or missing info
Before any agent processes documents, run them through a sanitization engine that strips malware, redacts PII, and checks for missing critical files. This prevents poisoned inputs and compliance violations from cascading through the system.
Too many steps for one model
Use a central orchestrator agent to break high-level goals into subtasks and delegate them to specialized workers. Pair fast, cheap models for extraction tasks with heavy reasoning models for complex evaluation to optimize cost and accuracy.
Workers need to view each other's live notes and policy docs
Combine short-term memory for live agent-to-agent data sharing with long-term vector memory for persistent domain knowledge. This lets workers instantly reference each other's outputs while grounding decisions in official guidelines.
Agents need to call external APIs without leaking credentials
Wrap external API calls in a standardized action layer that requests temporary, permission-mirroring tokens from an Identity and Access Management system. Revoke tokens immediately after use to prevent privilege escalation.
AI tries a high‑risk automatic decision
Place hard-coded deterministic rules between AI outputs and final execution. When a decision exceeds a risk threshold, automatically block the action and route it to a human expert for manual approval.
Requests pile up when many users chat with the agent
Synchronous LLM calls block the event loop, causing requests to queue up when multiple users interact with the agent. Converting node functions to async def and using ainvoke instead of invoke allows the agent to handle concurrent requests efficiently without blocking the main thread.
Need to call a LangGraph agent from other services
LangGraph workflows run locally by default, but production systems need a standardized interface. Wrapping the compiled graph in a FastAPI app creates a scalable HTTP endpoint that can accept JSON payloads, handle routing, and integrate with existing infrastructure.
My FastAPI app crashes after deployment
Dependencies and Python versions often differ between development and production environments. Packaging the agent in a Docker container ensures identical behavior anywhere by freezing the OS, Python version, and dependencies into a single image.
Want an always‑on API without vendor lock‑in
Self-hosting gives you full control and avoids vendor lock-in. A lightweight VPS provider can run Docker containers reliably, with Uvicorn handling high-concurrency requests via ASGI while the OS manages memory and networking.
RAG combines a vector store of document embeddings with an LLM prompt, letting the model answer using up‑to‑date information from PDFs or other sources. It works by searching for similar chunks, inserting them into the prompt, and asking the model to reason over that context.
Using `ingest_client.create_function` you register an async Python function, give it an ID and trigger name, then call it via the Ingest dev server. The framework captures input, output, errors, and execution time without extra code.
A small class wraps `quadrant_client`, handling collection creation, converting ID/vector/payload triples into `PointStruct`s for upserts, and performing a top‑k cosine similarity search that returns both text payloads and source IDs.
LlamaIndex’s PDFReader loads raw text, then a SentenceSplitter breaks it into chunks of configurable size (e.g., 1000 characters) with overlap (e.g., 200 chars) to preserve context across boundaries. The result is a list of strings ready for embedding.
The OpenAI `embeddings.create` endpoint accepts a list of strings and returns a matching list of float arrays. Using the same dimension (3072) as the Quadrant collection ensures compatibility for similarity search.
An automation follows a fixed sequence of steps, while an AI agent reasons, adapts, and chooses actions dynamically based on input. Recognizing this distinction helps you decide when to use a simple workflow or build a flexible agent.
Every AI agent is composed of three components: a large language model (brain) for reasoning, a memory store to retain context across interactions, and external tool integrations to act on the world. Understanding each part lets you design agents that can think, remember, and execute.
To connect an LLM like GPT‑4 Mini you need an OpenAI secret key from platform.openai.com. The key authenticates requests and is stored in n8n credentials, allowing the agent node to call the model.
n8n’s memory setting lets you choose a simple temporary store and define a context window (e.g., five messages). This determines how many prior interactions the LLM sees, enabling it to remember names or previous steps.
Each external service is added as a sub‑node under the AI agent node. Authentication is done via OAuth (Google) or API keys (OpenWeather). Once linked, the LLM can call functions like “read calendar events” or “send email”.
When a service lacks a built‑in integration you can create an HTTP request node, supply the full API URL (including your key), set method to GET, and enable JSON parsing. The agent then receives structured data it can reason over.
A well‑structured prompt tells the agent who it is, what goal to achieve, which data sources are available, any safety rules, and the desired format of the result. Using ChatGPT to generate this template saves time and ensures completeness.
If a node fails, you can screenshot the error, paste it into the chat with the LLM, and ask for step‑by‑step fixes. The model can suggest exact parameter changes, letting you iterate quickly.
When you want answers backed by your own documents
Retrieval Augmented Generation (RAG) adds context from a vector database before the LLM generates an answer, improving relevance and factuality. The pipeline consists of three steps: retrieve relevant documents, augment the prompt with those snippets, then generate the final response.
Need consistent answers from a language model
The temperature parameter controls randomness in token sampling: low values (<0.5) make outputs deterministic and reduce hallucinations, while higher values (>1) increase creativity. Adjusting temperature lets you match the needs of a production agent versus an exploratory prototype.
Turn a raw meeting transcript into tidy markdown notes
LangGraph represents agents as nodes and edges, enabling explicit state tracking across multiple reasoning steps. This graph‑based approach lets you orchestrate complex tasks like meeting‑note summarization with clear input/output flow.
Need the LLM to run my Python function
LangChain’s `@tool` decorator wraps ordinary Python functions so the LLM can invoke them during a reasoning loop, enabling dynamic data lookup or computation without leaving the agent context.
Getting info from a long transcript step by step
The react pattern splits an agent’s operation into a reasoning step (generate plan or query) followed by an action step (call a tool), looping until the task is complete. This makes agents capable of multi‑turn problem solving.
Open‑source stores like Chroma or FAISS run locally without cost, while managed services (Pinecone, Weaviate Cloud) offer scalability and automatic indexing. Selecting the right store balances budget, data size, and latency requirements.
Need quick meeting notes from a transcript
FastAPI provides automatic request validation, async handling, and interactive docs, allowing you to serve the agent for external applications with minimal code.
Can't tell why my meeting‑note bot stalls
LangSmith records each LLM call, tool invocation, and prompt/response pair, giving visibility into latency, token usage, and failure points, which is essential for production monitoring.
When a user asks with personal data
Embedding explicit instructions in the system prompt (e.g., “Never output personal data”) and using post‑generation filters reduces risk of exposing sensitive information from user inputs.
Need a ready‑to‑use AI agent without handling AWS setup
The starter toolkit CLI walks you through creating IAM roles, building a container with CodeBuild, pushing it to ECR, and deploying it to the Agent Core runtime. It automates all the infrastructure steps so you can get a production‑ready agent up in minutes.
My bot loses what I told it before
By importing the Agent Core memory SDK and configuring a SessionManager with a memory ID, actor ID, and session ID, agents can store conversational context for a single session and user‑wide preferences across sessions. The service manages scaling and isolation automatically.
Want to run custom code without security worries
Replacing a simple calculator tool with the Agent Core Code Interpreter gives your agent access to a managed sandbox where Python (or other language) code runs safely, without you managing containers or security patches.
Can't see if my AI agent is slow or failing
Agent Core provides a built‑in observability service with CloudWatch log aggregation and a web dashboard showing request latency, token usage, and tool invocation details, helping you debug and optimize your agents.
Agents drop or reorder steps
Instead of relying on prompt engineering to chain agents, use framework-level flow definitions like sequential or parallel agents. This works because LLMs are non-deterministic and often skip steps or reorder calls when given routing instructions, whereas predefined flows enforce strict execution graphs at the code level.
Too many tool definitions slow my AI
Replace broad tool definitions with skills, which are natural-language descriptions of specific workflows or domain expertise. This works because skills allow the agent to match user intent to the exact right process without loading unnecessary function signatures into the context window, cutting latency and token costs.
Unsure if your agent is degrading over time
Use a simulator to automatically run your agent against diverse, realistic scenarios and capture the inputs, outputs, and tool calls. This works because it creates a reusable golden dataset that lets you quantitatively measure performance drift, test framework updates, and catch regressions before they hit production.
Untracked agents everywhere
Deploy a centralized gateway that manages agent identities, maintains a registry of all deployed agents, and enforces access policies and authentication. This works because it creates a single choke point for governance, preventing untracked agents from accessing sensitive data or tools while enabling secure, scalable multi-agent communication.
Need answers from internal data
The Databricks Playground lets you create an LLM‑based agent that can call tools and query your company’s proprietary data, turning a generic model into a domain‑specific assistant. By attaching retrieval or API tools, the agent can fetch relevant information at runtime.
MLflow evaluate runs your agent against a labeled evaluation dataset and uses Databricks LLM judges to score relevance, correctness, and safety. The feedback highlights failing cases and suggests concrete tweaks, enabling an iterative improvement loop.
Want to track lineage and set policies for your AI bot
Registering your agent in Unity Catalog records its lineage, version, and access policies, then deploying it via a Mosaic AI model‑serving endpoint provides a managed API with traffic splitting, logging, rate limiting, and safety filters.
Need easy chat calls and want to know token usage
Direct SDK calls require boilerplate for authentication, client initialization, and response parsing. Using the official Python library standardizes the request format with system/user/assistant roles and provides a predictable response object structure that includes usage statistics. This abstraction lets you focus on prompt design rather than network handling.
Need to try a different LLM without touching code
Vendor SDKs lock you into specific APIs and response formats, making provider switching painful. LangChain provides a unified interface that normalizes prompts, model calls, and outputs across providers like OpenAI, Anthropic, and Google. This reduces boilerplate by up to 70% and enables instant A/B testing or cost balancing.
Want reliable AI replies in a set style
LLMs respond predictably when given explicit constraints, examples, or reasoning steps. Zero-shot relies on model knowledge, few-shot enforces style and format, and chain-of-thought forces step-by-step breakdowns for complex tasks. Structured prompting dramatically improves consistency and reduces hallucinations.
Keyword search returns nothing relevant
Keyword search fails when query phrasing differs from document wording. Embeddings convert text into high-dimensional vectors where semantic similarity maps to mathematical proximity, enabling meaning-based retrieval. Overlapping chunks preserve context across boundaries, and similarity thresholds filter out weak matches.
Need factual answers from private docs
Retrieval-Augmented Generation grounds LLM outputs in up-to-date, private data by injecting retrieved context into the prompt at runtime. This eliminates the need for model fine-tuning while preventing hallucinations. Strict system prompts that limit answers to retrieved context ensure factual accuracy.
When you need a branching, looping research flow that remembers results
Simple chains process data linearly, but production agents require branching, loops, and memory. LangGraph models workflows as a graph of nodes (functions) and edges (routing logic) that share a mutable state object. This enables conditional routing, iterative refinement, and persistent context tracking across complex tasks.
Claude EveryoneHaving to repeat project purpose and rules every time
Claude.md is a markdown file that Claude automatically reads at the start of every session in a project workspace. By writing project context, user profile, rules, and folder structure here you give the agent lasting guidance without re‑prompting each time.
Claude EveryoneWant the AI to show its step‑by‑step plan first
Planning mode forces the agent to present a written step‑by‑step plan and wait for your approval, preventing it from committing to wrong assumptions early. This small pause saves time correcting downstream errors.
Claude EveryoneNeed a full research report from a single topic
By writing a plain‑English SOP (research_agent.md) that tells Claude how to clarify scope, plan, research, synthesize, and save the output, you turn a single prompt into a repeatable autonomous agent for any topic.
Claude EveryoneOne long script needs bite‑size versions
A single SOP describing input location, content extraction steps, platform‑specific copy creation, and PDF conversion lets Claude automatically produce multiple deliverables from one source file, saving hours of manual repurposing.
Claude EveryoneWant to change only the executive summary in a report
Because Claude retains full project context, you can ask it to modify only a targeted part of a previously generated document (e.g., trim executive summary) without re‑creating the whole file, enabling fast incremental improvements.
Huge manuals and videos are a mess
Upload all unstructured documents (videos, PDFs) into Google NotebookLM and let it auto‑extract tables and summaries. This turns massive manuals into searchable structured data, cutting research time dramatically.
Need answers from only my company docs
A Gemini ‘gem’ lets you feed an entire NotebookLM plus extra files into a custom LLM that answers queries using only your proprietary data, providing a secure, domain‑focused retrieval‑augmented generation (RAG) layer.
Need separate AI helpers for each task
CrewAI’s low‑code studio lets you split a complex task into multiple specialized agents, each with a single responsibility. Clear role separation prevents hallucination caused by overlapping duties.
When your LLM returns unexpected JSON
Wrap your LLM calls with Pydantic models to automatically validate incoming data and outgoing responses, turning free‑form text into reliable JSON and catching malformed payloads early.
Need a bot that blocks prohibited queries
Using Unsloth’s PEFT LoRA adapters you can quickly teach a base HuggingFace model to refuse certain topics (e.g., financial advice) without retraining the whole model, keeping inference cheap and safe.
AI agents keep dead‑locking
LangGraph lets you draw a state graph where each node is an agent step and edges encode conditional hand‑offs, guaranteeing deterministic collaboration order and preventing dead‑ends.
When your chatbot only returns text
Integrating ElevenLabs’ text‑to‑speech API via LangChain’s community tools lets you convert any LLM response into a WAV file, creating a more natural conversational interface.
Want a simple web UI for your AI agent
Gradio provides a one‑file, zero‑config front end that can expose any Python function (including multi‑agent pipelines) as an interactive web interface with live logs.
Want to catch hallucinations, tool bugs and security flaws
Combine unit (prompt/tool), integration (full pipeline) and adversarial tests to catch hallucinations, tool failures, and security issues before deployment.
Want a multi‑step job done without watching
An agent loop consists of three stages—Observe, Think, Act—that repeat until the goal is achieved. The loop lets the LLM fetch context, plan next steps, and perform actions without human intervention.
Repeating brand info in every AI prompt
An agents.md file acts as a permanent system prompt that loads before every session, giving the agent role, business details, preferences, and tool instructions. This replaces ad‑hoc prompting and makes prompts short and reliable.
My chatbot never remembers my favorite color
memory.md is a mutable markdown file that agents update after each interaction to remember user preferences (e.g., favorite color, email sign‑off). By reading/writing this file each session, the agent simulates long‑term memory without hidden cloud state.
Want your LLM to update Gmail, calendar or Notion
Model‑Context Protocol (MCP) translates natural‑language calls from the LLM into API actions for external tools (Gmail, Calendar, Notion, etc.). By adding a connector in the harness UI, the agent can read/write those services without custom code.
Need to repeat a proposal process without re‑explaining it
A skill is a markdown file (.skill) that encodes a complete SOP: description, step‑by‑step instructions, and any reference assets. Once created, the agent can invoke the skill by name, instantly reproducing the exact process without re‑explaining it.
Need one spot to store AI agents, memory and skills across tools
Treat your local markdown hierarchy as an operating system: top‑level folders for each department (executive assistant, marketing, finance), each containing its own agents.md, memory.md, and a /skills subfolder. This structure lets you swap harnesses while preserving all knowledge.
Need to turn a vague objective into clear actions
An AI agent starts with a high‑level objective, then creates a detailed plan by decomposing the goal into smaller, manageable steps, similar to chain‑of‑thought prompting. This structured approach lets the agent know both what to achieve and how to proceed.
RAG lets an agent query external data sources (databases, websites, APIs) and inject the retrieved information into its language model responses, overcoming the static training cut‑off of base LLMs.
Want a bot that can talk to other services
Beyond text generation, agents can call tools such as web browsers, calendars, or custom APIs, allowing them to perform actions in the real world like sending emails or updating records.
Want an agent project set up for you
The langgraph new command bootstraps a complete agent project with pre-configured architecture, dependencies, and environment setup. It removes the friction of manual project initialization and ensures best practices are baked in from day one.
Want to change agent prompts on the fly
Running langgraph dev spins up a local server that LangSmith Studio connects to, enabling real-time debugging and trace viewing. Changes to prompts, tools, or architecture trigger automatic hot-reloads, allowing rapid iteration without restarting the server or redeploying.
Need a production‑ready AI agent without server work
langgraph deploy packages the agent, builds a containerized runtime, and provisions a hosted API endpoint on LangSmith. It handles infrastructure provisioning, versioning, and exposes REST endpoints and MCP protocol support out of the box.
Need to watch and control production agents from the terminal
The CLI provides a full lifecycle management suite (logs, list, delete) that keeps deployment operations terminal-native. Streaming logs and deployment status tracking eliminate the need to switch contexts between IDE and cloud UIs for routine ops.
The same set on /recipes, filtered by tool and role.
Shows how to construct production-ready AI agents using LangGraph, NotebookLM, and CrewAI
Shows how to create an AI agent by linking a language model to external data and orchestrating multi‑step workflows
Shows how to build an AI agent from scratch without coding
Shows how to build functional AI agents by creating reusable SOPs, implementing persistent preference storage, and applying context engineering
Agent Span moves the execution state of an AI agent from your local process to its own server, so the workflow keeps running even if your script or server restarts. It gives you a web UI where you can pause, resume, or inspect each step of the agent’s run. This makes long‑running or critical tasks more reliable.
First install the SDK with pip install agentspan, then launch the server using agent_span server start (default port 6767). When your process crashes, find the execution ID printed in the logs and run python resume_demo.py <execution_id> to continue from where it stopped.
Install Ollama (for example with brew install ollama on macOS), then pull the model using ollama pull llama3.1. Verify it’s available with ollama list, and configure Agent Span to call the local endpoint (e.g., http://localhost:11434) for inference. This avoids external API latency and cost.
RAG combines a vector store of document embeddings with an LLM prompt, letting the model answer using up‑to‑date information from PDFs or other sources. The system searches for text chunks similar to the query, inserts those chunks into the prompt, and asks the LLM to reason over that context. This gives you a clear mental model for building pipelines that retrieve relevant text and feed it to an LLM.
Gemini
Retrieval Augmented Generationingest_client.create_functionQuadrant client wrapperPointStructPDFReaderOpenAI embeddings.createAgent Spanagent_span server startOllamaagents.mdmemory.mdModel‑Context Protocol (MCP)Ask, share, or report — over on the Heidelberg AI community forum.