Fine-tuning results depend mostly on data quality. A few hundred excellent examples often beat thousands of mediocre ones.
Collect Representative Examples
Gather real inputs your application will see, covering common cases, edge cases and difficult ones.
Write High-Quality Outputs
Each output should be exactly what you want the model to produce: correct, well formatted and in the right style. Inconsistent outputs teach inconsistent behaviour.
Format
Most providers use conversation-style formats with system, user and assistant messages, typically in JSON Lines files. Follow the exact format your provider or library requires.
Clean
- Remove duplicates and near-duplicates.
- Fix errors and inconsistencies.
- Remove personal data and secrets.
- Balance categories where relevant.
Split
Keep a held-out test set, never used for training, to measure improvement honestly.
Start Small
Fine-tune on a small set, evaluate, find weaknesses, and add targeted examples.
Synthetic Examples
Language models can help draft examples, but review them carefully — they can introduce subtle errors and repetitive patterns.