Choosing Between a Small Model and a Large Hosted Model
Learn a practical framework for deciding when a small on-device model is the right architectural choice versus a large API-based model.
Start from the task, not the model size
The decision should start with the task's actual requirements: how broad is the input space, how much does quality on hard or unusual cases matter, what's the acceptable latency, and does the data need to stay on-device. A model-size decision made from those constraints will hold up better than one made from a general preference for 'bigger is better' or 'smaller is cheaper.'
Measuring the real quality gap
Don't assume a small model is 'good enough' or 'not good enough' — measure it on your actual task with your own eval set (see the evals topic), comparing a candidate small model against your current large-model baseline on the specific inputs your product handles. The gap on a narrow task is often much smaller than general benchmarks suggest, and sometimes larger, depending on how well the small model's training matches your domain.
Hybrid architectures: small model as a first pass
A common pattern uses a small, fast model to handle the majority of straightforward cases and routes only ambiguous or low-confidence cases to a larger model — capturing most of the latency and cost benefit of a small model while preserving quality on the hard tail. This requires a reliable confidence or routing signal, which is itself worth evaluating rather than assuming the small model 'knows when it doesn't know.'
Fine-tuning a small model for your domain
Because SLMs have less general capability to fall back on, fine-tuning or distillation from a larger model on your specific task tends to matter more for a small model's practical quality than for a large general model, which often does reasonably well with just good prompting. If you're choosing the small-model path, budget for a training or distillation step rather than expecting an off-the-shelf small model to match your domain out of the box.
Operational cost of the small-model path
Running your own small-model inference — on-device or at the edge — trades API cost for engineering ownership: packaging, updating, and testing a model shipped with your application, and handling the hardware diversity of real devices. Weigh this real operational cost against the per-request savings before assuming self-hosted is automatically cheaper.
Practical exercise
For one task you're evaluating, build a small eval set and score both a current large-model baseline and a candidate small model. Calculate the accuracy gap and the latency/cost difference, and write a one-paragraph recommendation for which is the right default — and under what condition you'd escalate to the other.
Sources and further reading
These primary or specialist references informed the concepts in this guide. Product details can change, so verify current documentation before implementation.