Model development involves many experiments. Without tracking, it's easy to lose the best result or be unable to explain how it was achieved.
What to Track for Every Run
- Code version (git commit).
- Data version and preprocessing steps.
- Hyperparameters and configuration.
- Metrics on training, validation and test sets, including by group.
- Artefacts: the trained model, plots, confusion matrices, evaluation reports.
- Environment: library versions and hardware.
- Notes: why the experiment was run and what was learned.
Tools
Experiment trackers such as MLflow and Weights & Biases log runs automatically and let you compare them side by side. Even a disciplined spreadsheet with links to saved artefacts is far better than nothing.
Good Habits
- Name experiments clearly, grouped by question.
- Log automatically from training code rather than by hand.
- Keep failed experiments too — they tell you what doesn't work.
- Compare with the baseline and with the current production model.
- Avoid repeatedly tuning against the test set; use it for final comparisons.
From Experiment to Production
Link the chosen experiment to the registered model, so the path from exploration to deployment is traceable.
For LLM Work
Track prompt versions, model names and settings, retrieval parameters and evaluation-set results the same way.