Serving language models efficiently is different from serving traditional models.
Why It's Hard
- Large models need lots of GPU memory.
- Generation is sequential, token by token.
- Request lengths vary widely.
Key Techniques
- Continuous batching: add and remove requests from batches as they start and finish, greatly improving throughput.
- KV caching: store attention keys and values from earlier tokens to avoid recomputation.
- Paged attention: manage cache memory efficiently.
- Quantisation: smaller, faster models.
- Speculative decoding: a small model drafts tokens that the large model verifies in parallel.
- Prefix caching: reuse computation for shared prompt prefixes.
Serving Frameworks
Open-source inference servers implement these techniques and expose standard APIs.
Metrics
- Time to first token.
- Tokens per second per request.
- Total throughput.
- Cost per million tokens.
Build or Buy
Managed APIs are simpler; self-hosting gives control and can be cheaper at scale, if you have the expertise.