Labelled data powers supervised learning and evaluation. Tools and workflows determine its quality.
Tool Types
- Open-source annotation tools for text, images, audio and video.
- Commercial platforms with workforce management.
- Managed labelling services.
- Custom interfaces for specialised tasks.
Choosing Tools
- Support for your data types and label formats.
- Collaboration, review and quality control features.
- Security and data residency.
- Integration with your data pipelines.
Workflow Design
- Guidelines: clear definitions with examples and edge cases.
- Pilot: label a small batch, measure agreement, refine guidelines.
- Production labelling with regular spot checks.
- Review: second-pass review of disagreements and difficult items.
- Feedback: update guidelines as new cases appear.
Model-Assisted Labelling
Pre-label with a model and have people correct it. Faster, but watch for people accepting wrong suggestions.
Labeller Welfare
Content moderation and sensitive material can harm labellers. Provide support, rotation and fair pay.