How to run LLMs in production: a 10-point checklist
A practical checklist for running LLMs in production safely: SLOs, observability, cost caps, prompt injection, fallbacks and runbooks. An SRE's view.
Shipping an LLM to production isn’t swapping one API call for another. You’re putting a non-deterministic, pay-per-request component that depends on an outside vendor in front of real users. Here’s the minimum an SRE team should require before the first user arrives.
1. Define what “working” means
Before writing code, write down your SLIs: success rate, p95 time to first token, total latency and a quality signal (for example, share of answers without complaints). Without them, you can’t tell when things get worse.
2. Set realistic SLOs
LLMs are slower and less stable than a database. An SLO of 99% of responses under 8 seconds may be more honest than 99.9% under 2 seconds. Tune it later, with data.
3. Log prompt, response and cost per request
Every call should emit an event with model, prompt version, input and output tokens, estimated cost, latency and outcome. Mask personal data before you store it.
4. Version your prompts
Prompts are code. Keep them in the repo, review them in PRs and tie each response to the version that produced it. When quality drops, you’ll want to know what changed.
5. Cap your spend
Limit tokens per request, requests per user and monthly spend, with an alert at 80% of budget. One bad loop can burn a month’s budget overnight.
6. Defend against prompt injection
Treat all retrieved content (documents, pages, emails) as data, never as instructions. Give the model only the tools it needs and require human approval for irreversible actions.
7. Plan for the vendor going down
Short timeouts, bounded retries, a circuit breaker and a useful degraded response (“I can’t answer right now — here’s a link”). If it’s critical, keep a second provider configured.
8. Evaluate before you ship
Build a set of real questions with expected answers and run it on every model or prompt change. That’s your regression test.
9. Alert on error budget burn
Fast- and slow-burn alerts warn you before users complain, without paging anyone for a one-off spike.
10. Write the runbook
What to do when latency spikes, when cost explodes or when the model starts answering wrong. A runbook turns a 3 a.m. incident into a list of steps.
Each of these items becomes a full chapter with code, dashboards and real examples in AI for SREs.
Put this into practice
AI for SREs
Your LLM returned HTTP 200. And it was wrong. Learn to run AI with SRE discipline.