Inference cost and latency often dominate the total cost of a machine learning system, especially for large models serving many users.
Measure First
Track latency percentiles, throughput and cost per prediction under realistic load before optimising.
Techniques
- Batching: process several requests together to use hardware efficiently; dynamic batching groups requests arriving close together.
- Caching: store results for repeated inputs, and cache expensive intermediate results such as embeddings.
- Quantisation: store weights with fewer bits (for example 8-bit or 4-bit), shrinking memory and speeding inference with a small accuracy cost.
- Distillation: train a smaller "student" model to mimic a larger "teacher".
- Pruning: remove weights or components that contribute little.
- Optimised runtimes: inference engines and compilers (such as ONNX Runtime or specialised LLM serving engines) are often much faster than training frameworks.
- Right-sized hardware: CPUs for small models, GPUs or specialised accelerators for large ones; autoscale with demand.
For Language Models
Techniques such as key–value caching, continuous batching, prompt caching and streaming responses matter greatly. Routing simple requests to smaller models also saves cost.
Always Re-Evaluate
Every optimisation can change outputs. Re-run your evaluation set and compare quality, not just speed.
Consider Batch Where Possible
If predictions don't need to be real time, precomputing them in batch is usually far cheaper.