ML systems need testing at several levels, like any software — plus some ML-specific checks.
Code Tests
Unit tests for feature engineering, preprocessing and post-processing logic.
Data Tests
- Schema: expected columns and types.
- Ranges and distributions.
- Missing values and duplicates.
- Freshness.
Model Behaviour Tests
- Invariance: predictions shouldn't change for irrelevant changes, such as names in a sentiment review.
- Directional expectations: increasing income shouldn't decrease a credit score, all else equal.
- Minimum functionality: simple cases the model must get right.
- Slice performance: acceptable results for important subgroups.
Integration Tests
End-to-end tests from input to prediction in a production-like environment, including latency.
Regression Tests
Compare new models against current models on fixed evaluation sets before release.
Automate
Run tests in CI/CD pipelines so every change is checked.