▸How-to Finance +1
Need a quick AI budget estimate
A defensible monthly AI budget you can put in front of the board — built from token math, not a guess.
Track 17 · Advanced
17.2
What AI really costs
Tokens, pricing, and the levers that cut your bill
1Overview
Every AI call is billed in tokens — and the bill is easy to cut once you can see it. Taught as a weak → strong ladder: understand what you pay (input vs output), trim per call (cap output, less context, right-size the model), then scale the savings with prompt caching, the Batch API and model routing. With worked cost math, persona examples and primary-sourced FAQs.
In this chapter you will learn how to quantify AI spending using token math and build defensible budgets that can be presented to stakeholders. You will gain practical methods for identifying cost drivers, selecting the cheapest model that meets quality thresholds, and estimating spend before running large jobs. The material also shows how to cut expenses by batching non‑urgent work, caching repeated prompts, limiting output length, using structured responses, and routing easy requests to cheaper models, so you can predict and control AI costs with confidence.
You're billed per token — in (what you send) and out (what the model writes), at roughly 4 characters per token. Output costs several times more than input, so that's where the money goes. This lab shows how to see your AI bill and cut it WITHOUT losing quality.
1.1After this chapter you can
→Read a bill in tokens — why output costs more, and estimate a job before you run it
→Cut spend per call: cap output, demand structured output, trim the context you send
→Apply the big levers — prompt caching, the Batch API, and model routing
→Measure and attribute spend so you know what is costing you what
1.2How are AI calls billed?
Each request is charged based on the number of tokens processed—both the tokens you send as input and the tokens the model returns as output.
1.3What’s the easiest way to lower each call’s cost?
Limit the output length, reduce unnecessary context in your prompt, and choose a smaller model that still meets accuracy needs, which together cut token usage per request.
1.4How can I save more when scaling up?
Use prompt caching, batch multiple queries with the Batch API, and route traffic to cheaper models for suitable tasks; these levers multiply savings across large volumes.
1.5The moves — weak → strong ladder
- 1Tokens are the unit of billing — and output is the expensive side
- 2Estimate the cost before you hit go
- 3Cap the output length
- 4Demand structured / JSON output
- 5Send less context (the cost lens on context-engineering)
- 6Right-size the model to the job
- 7Prompt caching — stop re-paying for the same prefix
- 8Batch API for anything that isn't urgent
- 9Model routing / cascades — cheap first, escalate rarely
- 10Measure & attribute your spend
2Techniques
Learn
Understand the bill
Know what you're paying for
Tokens are the unit of billing — and output is the expensive side AI isn't priced per message; it's priced per token — dollars per million tokens IN, and dollars per million tokens OUT. Those two numbers are not equal: the tokens the model WRITES cost several times more than the tokens it reads, so that's where the money is.
✕Instead of
Think of the cost as "roughly a few cents per message" and never look closer.
✓ Try this 💬 AI chat
Read the price as $/1M input and $/1M output (MTok). A token is ~4 characters (~0.75 words), so a page of text is ~500–700 tokens. Now you can see what each call actually costs.
✕Instead of
Treat input and output as one undifferentiated "usage" number.
✓ Try this 💬 AI chat
Know the split: output typically costs ~2–8× input (e.g. Claude Opus 4.8 ~$5 in / ~$25 out per MTok). Attack the dearest tokens first — cap or shorten what the model writes.
Why it works: You can't cut what you can't see. Anthropic's pricing page lists every model as a $/MTok input and $/MTok output pair, and counts a token as ≈4 characters / 0.75 words. Across providers the output rate is multiples of the input rate (Opus-class is ~5× on Anthropic) — so once you read the bill that way, the highest-leverage trims (capping length, demanding structured output) all act on the output side.
Estimate the cost before you hit go A big job's bill is predictable with grade-school arithmetic. Multiply tokens × rate × number of calls before you run it, not after.
✕Instead of
Kick off a 10,000-row job and find out what it cost on next month's invoice.
✓ Try this 💬 AI chat
Estimate first: expected (input + output) tokens × the $/MTok rate × N calls (remember ~4 chars ≈ 1 token). A 30-second calculation tells you if a "$4" job is really $400.
Why it works: A back-of-envelope estimate catches the order-of-magnitude surprises that hurt — the run you thought was a few dollars that is actually a few hundred. Estimating turns "hope it's cheap" into a number you decided on in advance.
Trim per call
Cheap wins on every request
Cap the output length The model will happily write three paragraphs where one sentence would do — and you pay for every word. Put a ceiling on it.
✕Instead of
Leave the answer length unbounded and ask an open-ended question.
✓ Try this 💬 AI chat
Set a max output limit (e.g. max_tokens) and ask for brevity in the prompt ("answer in one sentence", "≤5 bullet points"). Bound the most expensive side of the call.
Why it works: Output is the dear side, and an unbounded answer can 10× a call's output cost when the model rambles. A hard cap plus a "be concise" instruction is the single cheapest trim you can apply to every request.
Demand structured / JSON output Free-form prose is padded with connective words you are paying for. Ask for a tight schema and the output shrinks.
✕Instead of
Ask for "a nice write-up" and get prose full of "Certainly! Here is a detailed overview…".
✓ Try this 💬 AI chat
Ask for JSON or a fixed schema: {"category": "...", "amount": 0, "flag": true}. The reply carries the same information in far fewer tokens.
Why it works: Structured replies strip the prose scaffolding, cutting the expensive output side — and as a bonus they are machine-readable, so you skip a parsing step. Same answer, fewer written tokens, lower bill.
Send less context (the cost lens on context-engineering) Every turn re-bills the entire context you send. Pasting "everything, just in case" on every call is paying again and again for tokens the model barely uses.
✕Instead of
Paste the whole document, the full chat history and all your instructions on every single turn.
✓ Try this 💬 AI chat
Trim boilerplate, keep only the relevant slice, and summarize long history instead of resending it verbatim. (See the context-engineering chapter for the techniques — here the point is the cost.)
Why it works: Input is re-billed on every call, so a bloated context is a recurring tax, not a one-off. Sending less directly cuts the input side of every turn. (Cross-link: context-engineering covers compression and selection — this lab just adds the price tag.)
Right-size the model to the job Not every task needs the frontier model. Paying top-tier rates to label a sentiment or tidy a list is burning money on work a small model nails.
✕Instead of
Send a simple classification or reformatting task to the biggest, priciest model by default.
✓ Try this 💬 AI chat
Use a small/cheap model for the easy, high-volume work and reserve the frontier model for genuinely hard reasoning. (See the LLM index to pick.)
Why it works: A Haiku-class model can be roughly 5× cheaper than an Opus-class one, and for work it already handles well the quality is indistinguishable. Matching model size to task difficulty is free quality-neutral savings. (Cross-link: the LLM index.)
Scale the savings
The big structural levers
Prompt caching — stop re-paying for the same prefix If every call starts with the same big system prompt or reference document, you are paying full input price for those identical tokens every single time. Cache them once.
✕Instead of
Resend the same 50-page knowledge base / system prompt at full input price on every request.
✓ Try this 💬 AI chat
Cache the stable prefix. On Anthropic, cache READS bill at ~0.1× input (≈90% off); Google cached input is ≈90% off; OpenAI discounts cached input ~50–90% automatically on prefixes of ≥1,024 tokens.
Why it works: This is the biggest lever for any app with a fixed large preamble — a long system prompt, a document, a tool schema reused across calls. The cached portion bills at a fraction of full price (cache read 0.1× on Anthropic), so a heavy-prefix workload can drop most of its input cost.
Batch API for anything that isn't urgent If a job doesn't need an answer this second, don't pay live-traffic prices for it. Submit it asynchronously and take the discount.
✕Instead of
Run a 10,000-row overnight job through the real-time API at full price during the day.
✓ Try this 💬 AI chat
Submit it to the Batch API: ~50% off input AND output (Anthropic, OpenAI and Google all offer this), with results back within ~24h — most batches finish in under an hour.
Why it works: Batching halves the bill on anything that can wait — backfills, bulk summaries, nightly enrichment. The only trade-off is latency (asynchronous, up to ~24h), which for non-interactive work costs you nothing.
Model routing / cascades — cheap first, escalate rarely Sending every request to the frontier model is paying premium for the easy 80% that a cheap model would get right. Route by difficulty instead.
✕Instead of
Pipe every request, easy or hard, straight to the most expensive model.
✓ Try this 💬 AI chat
Try a cheap model first and escalate to the frontier model only when a confidence or keyword check says the case is hard (the FrugalGPT / RouteLLM pattern).
Why it works: Routing the easy majority to a cheap model can cut spend ~40–85% while holding ~95% of frontier quality (RouteLLM / FrugalGPT). The hard cases still get the big model — you just stop overpaying for the simple ones. (Text source: github.com/lm-sys/RouteLLM.)
Measure & attribute your spend You can't pull the right lever if you don't know where the money goes. Log tokens per request and tag spend by feature, model and user.
✕Instead of
Ship the feature and never look at the usage dashboard again until a bill spikes.
✓ Try this 💬 AI chat
Log input/output tokens per request, watch the provider usage dashboard, and tag spend by feature, model and user so you can see your most expensive surface.
Why it works: Attribution tells you which lever to pull next — the runaway feature, the model that's overkill, the user pattern that 10×s cost. Without it every other optimisation in this lab is a guess; with it you fix the expensive thing first.
3Lessons 7
3.1 Project your monthly AI cost on one page
A token‑based cost estimate that combines model rates, expected call volume and average tokens per request.
Produce a single‑page document that projects the monthly AI bill for your product
- Collect the model rate (price per 1 000 input and output tokens) you plan to use
- Calculate the average number of input and output tokens per API call for your feature
- Compute daily token usage by multiplying the averages by the expected call volume, then multiply by 30 days to obtain monthly totals
- Apply the model rate to the monthly token total to derive a dollar amount
- Create a one‑page table that records assumptions, token totals, the cost figure and a margin of error
- You'll see A concise spreadsheet‑style page listing assumptions, token totals and a clear monthly cost figure
- Takeaway Token math lets you turn vague usage ideas into concrete budget numbers before any code runs
- Check How do you combine model rates, expected call volume, and average tokens per request to calculate a projected monthly AI bill?
3.2 Set a maximum output length to cap token spend per call
A request parameter that limits how many tokens the model can generate in its response.
You will be able to reduce each API call’s cost by preventing overly long outputs.
- Open your AI service’s request console or code editor.
- Add a field named max_tokens (or the equivalent) to the JSON payload of your request.
- Set its value to a number that matches the longest response you actually need, such as 150 tokens.
- Send a test prompt and record the length of the returned text.
- Compare the token count with a call that omits the max_tokens field.
- You'll see The response stops at the specified length and uses fewer output tokens than an unrestricted call.
- Takeaway Limiting output directly controls variable costs because you only pay for the tokens you actually receive.
3.3 Cache a static SOP to reduce per‑call token usage
Prompt caching that stores a long, unchanging standard‑operating‑procedure (SOP) so only the variable part is sent on each request.
Modify an API workflow so the SOP is paid once and subsequent calls use about 10 % of its token cost
- Identify the large, static text block that appears in every prompt
- Create a cached version of this block using your provider’s caching mechanism or by storing it locally
- Update the request code to send only the dynamic portion while referencing the cached SOP
- Run two test calls: one with the full prompt and one with the cached approach
- Compare token usage reports from the API logs to verify the input‑token reduction
- You'll see The second call shows roughly a 90 % drop in input tokens compared to the uncached version
- Takeaway Separating immutable context from variable content lets you amortise large prompt costs across many calls
- Check What steps let you verify that caching a static SOP reduces input‑token usage by roughly 90 % compared to sending the full prompt each time?
3.4 Choose the cheapest model that meets your quality threshold
A comparison of different model tiers offered by an AI provider, each with its own price per token.
You will be able to select a lower‑priced model without sacrificing required answer quality.
- Identify at least two models from your provider’s pricing page (e.g., a standard and a smaller variant).
- Write a representative prompt that reflects the typical task you run.
- Run the prompt against each model and capture the full response text.
- Evaluate the answers using a simple checklist of must‑have criteria (accuracy, completeness, tone).
- Select the model that meets all criteria while having the lower per‑token price.
- You'll see A clear decision matrix showing which model satisfies your quality needs at the lower cost.
- Takeaway Model selection is a primary lever for cost reduction; cheaper models often suffice for many routine tasks.
3.5 Implement prompt caching to reuse identical requests
A local key‑value store that saves the response of a given prompt so future calls can retrieve it instead of invoking the API.
You will be able to eliminate token charges for repeated prompts by serving cached answers.
- Create a simple JSON file or dictionary named prompt_cache.json.
- Write code that, before calling the AI API, checks if the exact prompt string exists as a key in the cache.
- If found, load and return the stored response; otherwise, call the API.
- After receiving a new response, add an entry to the cache with the prompt as the key and the response text as the value.
- Run the same prompt twice and note that the second run does not trigger an API request.
- You'll see The second execution returns instantly from the local file and no new tokens are billed.
- Takeaway Caching eliminates redundant token usage, turning repeat queries into free look‑ups.
3.6 Run a 10 000‑record classification job with the Batch API at half cost
An asynchronous batch execution that sends thousands of records in one request, trading latency for a lower per‑token price.
Submit a bulk classification job for 10 000 items using the Batch API and confirm the cost is about 50 % cheaper than individual calls
- Prepare a CSV or JSON file containing the 10 000 records to classify
- Create a batch request payload that references your model, includes any needed prompt template, and points to the input file
- Submit the batch job via the provider’s Batch API endpoint and note the estimated token cost returned in the response
- Retrieve the results file after completion and record the actual token usage reported
- Calculate the effective per‑record cost and compare it to a baseline where each record is processed with separate real‑time calls
- You'll see The batch job finishes with an invoice showing roughly half the total tokens (and dollars) of the equivalent real‑time run
- Takeaway Batching non‑urgent workloads can dramatically reduce token pricing while still delivering correct results, especially for large volumes
- Check How can you demonstrate that processing 10 000 records with the Batch API yields about a 50 % lower per‑record token cost than handling them individually?
3.7 Enable AI Gateway spend limits to prevent runaway bills
A configuration option in Cloudflare’s AI Gateway that caps the total amount you can be charged for AI usage.
You will be able to set a hard ceiling on monthly AI spending so you never exceed budgeted costs.
- Log into your Cloudflare dashboard and navigate to the AI Gateway section.
- Locate the Spend Limits or similar setting in the configuration panel.
- Enter a dollar amount that matches your maximum acceptable monthly spend, such as $50.
- Save the changes and confirm that the limit is active by checking the status indicator.
- Trigger an API call that would exceed the limit and verify that the request is blocked or throttled.
- You'll see Requests that would push total charges beyond the set amount are rejected, protecting you from unexpected fees.
- Takeaway Spend limits act as a safety net, ensuring cost control even when usage spikes unexpectedly.
4You’ll know it worked 26 checkable outcomes in this chapter
- ✓Board members approve the AI budget after reviewing the token-based estimate
- ✓Job completes overnight and cost is roughly half of real-time pricing
- ✓80% of tickets are answered by the cheap model and 20% are escalated to the frontier model
- ✓A single number appears showing the estimated monthly cost
- ✓Spend stays below the defined ceiling during launch week
- ✓You see a calculated budget amount and a savings figure from caching
- ✓Thousands of papers are filtered by the small model, and only the complex ones are sent to the frontier model for detailed reasoning
- ✓The monthly cost estimate shows a lower figure after enabling policy caching
26 outcomes in all — one per recipe below.
5FAQ, Tips & How-to 38
one problem, one solution, one action
Dashboards & analytics6
▸How-to Finance +1
You can see which feature is the cost runaway and aim your optimisation there instead of guessing.
▸How-to Finance +1
Which AI model is the cheapest that still meets my quality requirements?
A model choice backed by a number — the cheapest model that still clears the bar, with the saving quantified.
▸How-to Founder +1
A cost-per-user number that tells you whether the pricing or the free tier is sustainable.
▸How-to Founder +1
Don’t know how much an AI feature will cost at launch
A launch you've cost-modelled in advance, with the trimming levers chosen before users arrive — not after a bill shock.
▸How-to Physician
A defensible monthly AI line-item for the practice budget, plus a model choice backed by an accuracy check rather than "biggest model available" — with cost and clinical-safety trade-offs made explicit up front.
Internal tools & ops5
▸How-to Operations +1
Want to classify 10 000 records with no rush
The same job done overnight at roughly half the cost — latency you don't care about traded for a discount you do.
▸How-to Operations +1
Repeating a long SOP in every request
The repeated SOP stops being re-billed at full price on every call — roughly 90% off that fixed chunk.
▸How-to Operations +1
No invoice surprises — you decide the spend in advance instead of discovering it after.
▸How-to Founder +1
Nightly job that can wait for morning results
A clear call on whether to batch the nightly job — usually yes, for half the cost and no UX impact.
▸How-to HR / People +1
A rollout cost you've modelled in advance, with the policy-caching saving already counted.
Customer & client portals2
▸How-to Support +1
Easy tickets waste money on a big model
The easy majority handled cheaply, the hard minority still handled well — big savings at near-frontier quality.
▸How-to Support +1
Replies are too long and unstructured
Shorter, structured replies that cost less per answer and are easier to render — same help, fewer written tokens.
Knowledge & docs1
▸How-to Support +1
Support bot reloads help‑centre articles on each question
A grounded support bot whose fixed knowledge base stops being re-billed on every question.
Trackers2
▸How-to Small biz +1
A clear monthly figure so you can decide if the AI feature is worth it before it's live.
▸How-to Small biz +1
Sending whole doc and chat log every time
Lower cost on every call of a workflow you run constantly — without losing the answer quality.
Content & marketing5
▸How-to Small biz +1
Bulk product copy is cheap‑priced
The same bulk drafting at a fraction of the cost, with the frontier model saved for the few hard pieces.
▸How-to HR / People +1
Job descriptions get too long
Consistent, right-length JDs that also cost less per draft because the output is bounded.
▸How-to Creator +1
Pay for the brand guide only once and limit draft size
On-brand drafts where the shared style guide is billed once and each draft's output is bounded — both cost sides trimmed.
▸How-to Creator +1
Too pricey bulk captions
The bulk content stream produced ~5× cheaper, with quality held where it matters.
▸How-to Creator +1
A content-calendar cost you set in advance, with the cheap-model and output-cap savings folded in.
CRM & sales3
▸How-to Sales +1
Product details repeat in every email and cost tokens
Personalised emails at scale where the shared product context is billed once, not on every send.
▸How-to Sales +1
Outreach emails get too long and pricey
Crisper emails that also cost less to generate — a quality and a cost win from the same cap.
▸How-to Sales +1
A campaign budget you set in advance, with the caching saving already factored in.
Research & data tools3
▸How-to Scientist +1
Need to price a 5,000-paper run
A large literature pass you've priced in advance and run at roughly half cost.
▸How-to Scientist +1
Every query repeats a long protocol text
The reused protocol stops being re-billed on every query — ~90% off that fixed context.
▸How-to Scientist +1
Sorting thousands of paper abstracts cheaply
Thousands of papers triaged cheaply, with the expensive model spent only where the reasoning is hard.
Forms, surveys & feedback1
▸How-to HR / People +1
Many survey responses to summarize without rush
All responses summarised at half the cost, ready when you arrive — no live-traffic premium paid.
▸How-to Everyone
Non‑urgent jobs that can wait
If a job doesn't need an answer this second, don't pay live-traffic prices for it. Submit it asynchronously and take the discount. Batching halves the bill on anything that can wait — backfills, bulk summaries, nightly enrichment. The only trade-off is latency (asynchronous, up to ~24h), which for non-interactive work costs you nothing.
▸How-to Everyone
Can't tell which part of your app is costing you
You can't pull the right lever if you don't know where the money goes. Log tokens per request and tag spend by feature, model and user. Attribution tells you which lever to pull next — the runaway feature, the model that's overkill, the user pattern that 10×s cost. Without it every other optimisation in this lab is a guess; with it you fix the expensive thing first.
▸How-to Everyone
Paying premium for easy queries
Sending every request to the frontier model is paying premium for the easy 80% that a cheap model would get right. Route by difficulty instead. Routing the easy majority to a cheap model can cut spend ~40–85% while holding ~95% of frontier quality (RouteLLM / FrugalGPT). The hard cases still get the big model — you just stop overpaying for the simple ones. (Text source: github.com/lm-sys/RouteLLM.)
▸How-to Everyone
Every request repeats a long system prompt
If every call starts with the same big system prompt or reference document, you are paying full input price for those identical tokens every single time. Cache them once. This is the biggest lever for any app with a fixed large preamble — a long system prompt, a document, a tool schema reused across calls. The cached portion bills at a fraction of full price (cache read 0.1× on Anthropic), so a heavy-prefix workload can drop most of its input cost.
▸How-to Everyone
Long AI replies cost more
The model will happily write three paragraphs where one sentence would do — and you pay for every word. Put a ceiling on it. Output is the dear side, and an unbounded answer can 10× a call's output cost when the model rambles. A hard cap plus a "be concise" instruction is the single cheapest trim you can apply to every request.
▸How-to Everyone
Copy‑pasting the entire conversation on each request
Every turn re-bills the entire context you send. Pasting "everything, just in case" on every call is paying again and again for tokens the model barely uses. Input is re-billed on every call, so a bloated context is a recurring tax, not a one-off. Sending less directly cuts the input side of every turn. (Cross-link: context-engineering covers compression and selection — this lab just adds the price tag.)
▸How-to Everyone
Sending all simple classifications to the biggest model
Not every task needs the frontier model. Paying top-tier rates to label a sentiment or tidy a list is burning money on work a small model nails. A Haiku-class model can be roughly 5× cheaper than an Opus-class one, and for work it already handles well the quality is indistinguishable. Matching model size to task difficulty is free quality-neutral savings. (Cross-link: the LLM index.)
▸How-to Everyone
Free-form AI prose adds extra tokens you're paying for
Free-form prose is padded with connective words you are paying for. Ask for a tight schema and the output shrinks. Structured replies strip the prose scaffolding, cutting the expensive output side — and as a bonus they are machine-readable, so you skip a parsing step. Same answer, fewer written tokens, lower bill.
▸How-to Everyone
A big job's bill is predictable with grade-school arithmetic. Multiply tokens × rate × number of calls before you run it, not after. A back-of-envelope estimate catches the order-of-magnitude surprises that hurt — the run you thought was a few dollars that is actually a few hundred. Estimating turns "hope it's cheap" into a number you decided on in advance.
▸How-to Everyone
AI isn't priced per message; it's priced per token — dollars per million tokens IN, and dollars per million tokens OUT. Those two numbers are not equal: the tokens the model WRITES cost several times more than the tokens it reads, so that's where the money is. You can't cut what you can't see. Anthropic's pricing page lists every model as a $/MTok input and $/MTok output pair, and counts a token as ≈4 characters / 0.75 words. Across providers the output rate is multiples of the input rate (Opus-class is ~5× on Anthropic) — so once you read the bill that way, the highest-leverage trims (capping length, demanding structured output) all act on the output side.
The same set on /recipes, filtered by tool and role.
6Videos 2
Spring AI Prompt Caching: Stop Wasting Money on Repeated Tokens
A hands-on look at prompt caching — why re-sending the same prefix wastes money and how caching the stable part cuts the input bill.
Is This the End of RAG? Anthropic's NEW Prompt Caching
A beginner-friendly intro to Anthropic prompt caching and what the ~90%-off cache read changes about how you structure prompts.
7FAQ 8
Why does output cost more than input?
Because generating tokens is more expensive for the provider than reading them, every major price list charges a higher rate for output than input — often several times higher (an Opus-class model is roughly $5 per million input vs $25 per million output). The practical upshot: the cheapest wins come from trimming what the model WRITES — cap the length, ask for structured/JSON output — not just what you send it.
What is a token, and how do I estimate a job's cost?
A token is a chunk of text the model bills on — roughly 4 characters, or about 0.75 words, so ~750 words is ~1,000 tokens. To estimate a job: count the expected input and output tokens for one call, add them, multiply by the model's $/1M rate, and multiply by the number of calls. That 30-second sum is enough to catch a job you thought cost a few dollars that actually costs a few hundred.
How much does prompt caching actually save?
A lot, on the cached portion. On Anthropic, reading from the cache bills at 0.1× the normal input rate — about 90% off — while writing to the cache costs a one-time 1.25× (5-minute) or 2× (1-hour) the input rate. So for a large prefix you reuse many times, you pay the small write premium once and then ~10% of input price on every later call. The bigger and more-reused the fixed prefix, the bigger the win.
When does caching pay off?
When the same large chunk of tokens leads many of your calls — a long system prompt, a reference document, a big tool schema, a few-shot block. You pay a one-time write premium (1.25× or 2× input) to put it in the cache, then read it back at 0.1×, so it pays off as soon as the prefix is reused even a few times. It does NOT help one-off prompts or calls whose front matter changes every time.
What's the Batch API discount, and what's the trade-off?
About 50% off — on both input and output — in exchange for asynchronous, non-real-time processing. You submit a batch of requests and get results back within 24 hours, though in practice most batches finish in under an hour. The only cost is latency, so it's ideal for anything that doesn't need an instant answer: bulk summaries, backfills, nightly enrichment, evals.
Do OpenAI and Google also discount caching and batching?
Yes. Google's Gemini API offers context caching (~90% off the cached input) and a Batch Mode at 50% off. OpenAI applies prompt caching automatically to prefixes of ≥1,024 tokens (discounting cached input ~50–90%) and offers a Batch API at 50% off with results within ~24h. The exact percentages differ by provider, but the two big structural levers — caching a fixed prefix and batching non-urgent work — exist everywhere.
Can I stack the caching and batch discounts?
They are independent levers that act on different parts of the bill: caching cuts the price of the repeated input prefix, batching halves what remains (input and output) in exchange for latency, and a right-sized model lowers the base rate underneath both. On a large non-urgent job with a fixed prefix you can apply all three for compounding savings — check your provider's docs for the exact interaction in your account.
What's the cheapest way to process thousands of items?
Combine the levers: pick the cheapest model that still passes your quality bar (often a small/Haiku-class one for bulk work), cache any fixed prefix that every item shares, and submit the whole run through the Batch API for ~50% off. Estimate the token cost first so there are no surprises. For a fixed-prefix, non-urgent bulk job, batch + a cheap model is the cheapest path.
8Glossary 7 terms
Show the 7 terms
Pricing
$/1M tokens (MTok)- How AI is priced: dollars per million tokens. Always two numbers — one for input (tokens read) and one for output (tokens written).
Input vs output tokens- Tokens you send (input) vs tokens the model writes (output). Output usually costs 2–8× input, so it is the side worth trimming first.
Token estimate- A quick size guess: ~4 characters ≈ 1 token (~0.75 words). Multiply expected tokens × rate × number of calls to price a job before running it.
Caching
Prompt caching- Reusing a stored copy of a fixed prompt prefix so you don't re-pay full input price for it each call. On Anthropic, cache reads bill at ~0.1× input (~90% off).
Cache hit vs cache write- A cache write stores the prefix (one-time 1.25×/2× input premium); a cache hit reads it back cheaply (0.1× input on Anthropic). You pay the write once, then hits forever.
Levers
Batch API- Submit requests asynchronously for ~50% off input and output; results within ~24h (often under an hour). For work that doesn't need a live answer.
Model routing / cascade- Send each request to a cheap model first and escalate to the frontier model only on hard cases (FrugalGPT / RouteLLM) — big savings at near-frontier quality.
9See also
💬 Discuss this chapter
Ask, share, or report — over on the Heidelberg AI community forum.