Training neural networks involves many choices. Some techniques are standard because they reliably help.
Optimisers
Adam and its variant AdamW are common defaults, adapting learning rates per parameter. Stochastic gradient descent with momentum remains competitive for some tasks.
Learning Rate Schedules
- Warm-up: start with a small learning rate and increase it.
- Decay: reduce the rate over training, for example with cosine schedules.
Normalisation
Batch normalisation and layer normalisation stabilise activations, allowing faster, more reliable training. Transformers typically use layer normalisation.
Regularisation
- Dropout: randomly disable units during training.
- Weight decay: penalise large weights.
- Data augmentation: create varied training examples.
- Early stopping: stop when validation performance stops improving.
Initialisation
Sensible weight initialisation schemes prevent signals from vanishing or exploding in deep networks.
Monitoring
Track training and validation loss, gradient norms and learning rate. Sudden spikes or flat lines signal problems.
Start From Pretrained Models
For most practical tasks, fine-tuning a pretrained model beats training from scratch.