Models can leak information about the data they were trained on. This matters when training data includes personal or confidential information.
Membership Inference
Determining whether a specific record was in the training set — for example, whether someone's medical record was used — by analysing the model's responses.
Training Data Extraction
Language models can memorise and reproduce rare sequences from training data, such as names, email addresses, phone numbers or code, especially if they appear multiple times.
Model Inversion
Reconstructing representative features of training data from a model, such as approximate faces from a face recognition model.
Attribute Inference
Deducing sensitive attributes about individuals from model outputs.
Defences
- Data minimisation: don't train on personal data you don't need.
- Deduplication and scrubbing: remove repeated and sensitive records.
- Differential privacy: training techniques that limit how much any one record influences the model.
- Output filtering: detect and block personal data in outputs.
- Access controls and rate limits.
Legal Context
Privacy laws may treat models trained on personal data as processing that data. Involve privacy specialists early.