Deploying to Production
Prepare LLM applications for reliable releases, monitoring, and incident response.
Separate the moving parts
Keep provider clients, prompts, retrieval, business rules, validation, tools, and presentation logic behind clear interfaces. This allows a model or provider to change without rewriting the product. Define which components are stateless and which hold data. A modular boundary also makes it easier to test failures locally instead of discovering them only after deployment.
Configuration and secrets
Use environment-specific configuration for model IDs, endpoints, limits, feature flags, and retention settings. Store credentials in a secret manager or protected deployment configuration, never in source or prompts. Validate required settings at startup without logging secret values. Make the active configuration visible to operators through safe metadata so incidents do not depend on guessing which model is running.
set -eu
curl --fail --silent https://example.com/health
curl --fail --silent -X POST https://example.com/smoke-test \
-H 'Content-Type: application/json' \
-d '{"prompt":"Return the word ready","dry_run":true}'Release safety
Use automated tests, evaluation gates, staging, canary traffic, feature flags, and a rollback plan. Test provider errors, streaming, timeouts, retries, tool authorization, empty retrieval, and malformed model output. Database or index migrations need compatibility checks with both old and new application versions. A deployment is not safe merely because the process exits successfully.
Reliability patterns
Set connection and total-operation deadlines, bound retries, use circuit breakers, and return honest degraded responses when dependencies fail. Queue long jobs and make them idempotent so a retry does not duplicate an email, payment, or record. Separate interactive traffic from batch work. Define health checks that distinguish process availability from useful model and dependency readiness.
Monitoring and traces
Monitor request volume, latency stages, errors, token usage, cost, model versions, validation failures, safety events, and user feedback. Use correlation IDs across retrieval, tools, and provider calls. Redact prompts and outputs by default, with controlled access for investigations. Dashboards should support a question such as “what changed for this failing request?” rather than only display totals.
Incident response
Prepare runbooks for provider outage, leaked data, unsafe output, runaway spend, retrieval corruption, and model regression. Assign owners, escalation paths, communication templates, and evidence-retention rules. During an incident, preserve enough trace information to understand impact while limiting further exposure. Practice the fallback and rollback path before it is needed under pressure.
Post-release learning
Review quality, reliability, cost, and user feedback after deployment, then compare them with the release hypothesis. Record what improved, what regressed, and which follow-up test or control is needed. Update the evaluation set with representative failures. Production operation is a feedback loop; the release is only successful when the system remains understandable and maintainable over time.