A model only creates value when its predictions reach people or systems. How you deploy depends on how predictions are used.
Deployment Patterns
- Batch scoring: run the model on a schedule and store predictions in a table — nightly churn scores, weekly demand forecasts. Simple and robust.
- Real-time API: a service returns predictions on request, within milliseconds to seconds — fraud checks during payment, recommendations on page load.
- Streaming: score events as they flow through a stream.
- Embedded / on-device: run in a mobile app, browser or edge device for low latency, offline use or privacy.
Packaging
Package the model with its preprocessing and exact dependencies — often in a container — so production behaves like training. Formats such as ONNX help run models across frameworks.
Safe Rollout
- Shadow mode: the new model runs alongside the old one without affecting decisions, so outputs can be compared.
- Canary release: send a small share of traffic to the new model first.
- A/B tests to measure business impact.
- Rollback to the previous version in one step.
Operational Concerns
Latency, throughput, scaling, cost per prediction, input validation, logging for debugging and audits, and security.
Fallbacks
Decide what happens when the model is unavailable or uncertain: default rules, cached predictions or human review.
Choose the Simplest Pattern
If predictions can be computed in advance, batch scoring avoids much of the complexity of real-time serving.