Models become stale as the world changes. Retraining keeps them useful, but it must be done carefully.
When to Retrain
- Scheduled: at fixed intervals (weekly, monthly). Simple and predictable; may retrain unnecessarily or too late.
- Triggered: when monitoring detects drift or performance drops. Efficient but depends on good monitoring.
- Continuous / online: update incrementally as new data arrives. Fast-adapting but harder to validate and more vulnerable to bad data.
Choosing a Window of Data
- All history: stable, but may dilute recent patterns.
- Recent window: adapts faster to change, but may forget rare or seasonal patterns.
- Weighted: give more weight to recent data.
Test options on historical data: how would each strategy have performed over the past year?
Safe Retraining
- Validate new training data before training.
- Train with the same reproducible pipeline.
- Evaluate on recent held-out data and compare with the current production model.
- Check fairness and edge-case tests.
- Deploy gradually, with rollback available.
Beware Feedback Loops
A model's own predictions can shape future training data — fraud flagged as risky is investigated more, so more fraud is found there. Account for this when constructing labels.
Don't Retrain Blindly
If performance dropped because of a broken data pipeline, retraining on bad data makes things worse. Diagnose first.
Record Every Version
Keep each retrained model's data window, metrics and approval, so changes in behaviour can be traced.