API Cost and Latency Management
Control usage while keeping AI features responsive and useful.
Understand the cost equation
AI cost is driven by input tokens, output tokens, model price, retries, tool calls, concurrency, and sometimes image or audio size. Aggregate spend can hide an expensive user journey with multiple failed attempts. Measure cost per successful task and per active user, not only cost per API request. Include preprocessing, storage, and observability costs when comparing architectures.
Measure latency correctly
Users experience connection time, queue time, time to first token, generation time, tool delays, and final rendering. Capture these stages separately so an optimization targets the real bottleneck. Streaming can improve perceived responsiveness without reducing total work. Set deadlines for the complete workflow and communicate progress when a long operation cannot reasonably be made instant.
input_tokens, output_tokens = 1200, 300
input_rate, output_rate = 0.000003, 0.000015
cost = input_tokens * input_rate + output_tokens * output_rate
print(f"estimated_cost_usd={cost:.5f}")Use the right model
Route classification, extraction, rewriting, and simple summarization to a smaller model when evaluation shows it is sufficient. Reserve a larger model for tasks that benefit from its capability. Routing should be explicit and monitored because a cheap model that causes retries or human corrections may cost more overall. Keep a fallback path for provider or model unavailability.
Reduce unnecessary context
Long instructions, duplicated history, irrelevant retrieval chunks, and verbose tool results consume tokens and slow generation. Summarize older turns, retrieve selectively, compress structured data, and set clear output limits. Do not remove evidence merely to save money; measure whether context reduction harms groundedness or completeness. Token budgets should reflect the task contract rather than an arbitrary global maximum.
Caching and batching
Cache stable embeddings, repeated retrieval results, and deterministic transformations when their inputs and permissions match. Never share a response across users without including all relevant authorization context in the cache key. Batch offline jobs and independent requests where the provider supports it. Streaming and caching need careful invalidation, because a fast stale answer is still a product failure.
Guardrails for spend
Apply per-user and per-tenant quotas, request size limits, concurrency caps, and maximum workflow budgets. Use exponential backoff only for transient failures and avoid retrying invalid requests. Add alerts for spend rate, token growth, retry storms, and unexpected model routing. Give operators a kill switch or degraded mode before a runaway loop becomes a large invoice.
Optimize with evidence
Change one cost or latency variable at a time and replay a representative evaluation set. Compare quality, time to first token, completion time, peak concurrency, error rate, and successful-task cost. Document the chosen trade-off and its operating limits. Optimization is complete when the product meets its user promise sustainably, not when one synthetic benchmark becomes faster.