CI/CD automates testing and releasing software. Machine learning adds data and trained models, which need their own automated checks.
Continuous Integration
On every change:
- Code tests: unit tests for feature engineering and data processing.
- Data validation: schema and quality checks on training data.
- Training smoke test: train on a small sample to catch errors early.
- Style and security checks.
Continuous Training
When data or code changes (or on a schedule), a pipeline retrains the model reproducibly and records the results.
Evaluation Gates
Before a new model can be deployed, it must:
- meet minimum performance thresholds;
- perform at least as well as the current production model on the same test set;
- pass fairness and robustness checks;
- stay within latency and size limits.
Continuous Delivery
Deploy approved models automatically to staging, run integration tests, then release through shadow, canary or staged rollouts with automatic rollback on problems.
For LLM Applications
Prompts, retrieval configurations and tool definitions are code too. Run evaluation test sets automatically whenever they change, and block releases that regress on critical cases.
Keep Humans in the Loop Where It Matters
High-impact models may require human review of evaluation reports before promotion, even with automated gates.
Start Small
Automate the steps that break most often first — usually data validation and evaluation — and expand over time.