High-Throughput Serving
Understand batching, concurrency, and hardware choices for serving many local model requests.
Throughput versus latency
Throughput measures completed work over time, while latency measures how long one request waits. A single interactive user usually values fast time to first token; a batch job may prefer maximum tokens per second. Serving design should state which metric matters and for what traffic shape. Optimizing one can harm the other, especially when batching makes individual requests wait for more work.
Continuous batching
A server can combine sequences that arrive at different times and schedule their next tokens together. Continuous batching keeps hardware busy as requests finish, unlike a simple batch that waits for every sequence. It requires careful scheduling, memory accounting, cancellation, and fairness. The benefit depends on prompt lengths, output lengths, concurrency, and whether the runtime has efficient attention kernels.
server:
max_concurrency: 8
max_queue_size: 32
max_context_tokens: 4096
reject_when_queue_full: trueKV-cache management
During generation, the model stores attention keys and values so it does not recompute the full history for every token. This cache can consume substantial memory, especially for long contexts and many simultaneous users. Paged or block-based allocation reduces fragmentation and makes scheduling more flexible. Set maximum context and output limits, then observe cache pressure under realistic traffic.
Hardware and parallelism
CPU serving is simple and may be economical for small models or low traffic. GPUs improve parallel matrix operations, while multiple devices can distribute weights or requests. Hardware support varies by runtime and precision. Benchmark the complete stack, including data transfer and startup time, rather than comparing theoretical compute alone. A modest device with a well-matched model can beat an expensive but poorly configured setup.
Queues and backpressure
An unbounded request queue turns overload into rising latency and eventual memory exhaustion. Use bounded queues, admission control, per-user limits, and clear rejection responses. Separate interactive traffic from offline jobs when possible. Backpressure should be visible in metrics and logs, and clients should receive retry guidance only when retrying is safe and likely to succeed.
Observability
Track request count, queue time, time to first token, generation time, tokens per second, input and output lengths, cancellations, errors, memory use, and model load time. Break metrics down by model and route. Avoid logging sensitive prompt content by default. These measurements reveal whether a problem comes from scheduling, model size, context growth, hardware saturation, or a slow client.
Capacity planning
Create a workload model with requests per minute, concurrency, token distributions, peak periods, and acceptable latency. Load-test with representative prompts and cancellations, then identify the point where quality or responsiveness degrades. Reserve headroom for failures and maintenance. Scale by reducing context, selecting a smaller model, adding devices, or distributing replicas only after measuring the actual bottleneck.