Training large neural networks from scratch needs enormous data and compute. Transfer learning reuses what a model has already learned.
The Idea
A model pretrained on a large, general dataset — millions of images, or vast amounts of text — learns broadly useful features: edges and textures, or grammar and meaning. Those features are valuable for many other tasks.
Three Ways to Reuse a Model
- Use it as is: if the pretrained task matches yours, such as general sentiment classification.
- Feature extraction: remove the final layer and use the model's internal representations (embeddings) as features for a small new model.
- Fine-tuning: replace the final layer and continue training on your data, usually with a low learning rate.
How Much Data You Need
Feature extraction can work with a few hundred labelled examples. Fine-tuning typically needs hundreds to thousands. Training from scratch needs far more.
Tips
- Pick a pretrained model whose original domain is close to yours.
- Use the same preprocessing the model was trained with.
- Freeze early layers first and unfreeze gradually if needed.
- Watch for overfitting on small datasets.
Where to Find Pretrained Models
Model hubs host thousands of open models with model cards describing their training data, licences and limitations. glitchdata's hub references several, such as ResNet-50 and all-MiniLM-L6-v2.