Well-designed pipelines are reliable, maintainable and easy to recover.
Idempotency
Running a pipeline twice for the same period should produce the same result, not duplicates. Use overwrites of partitions or merge operations rather than blind appends.
Incremental Loading
Process only new or changed data, using timestamps, change data capture or watermarks. Faster and cheaper than full reloads.
Backfills
Design pipelines to reprocess historical periods easily, parameterised by date.
Layered Architecture
- Raw (bronze): data as received.
- Cleaned (silver): validated, standardised.
- Curated (gold): business-ready tables and metrics.
Keeping raw data allows reprocessing when logic changes.
Late-Arriving Data
Plan for records arriving after their period has been processed; reprocess recent windows.
Dependency Management
Orchestrators run steps in order and retry failures.
Observability
Log row counts, durations and data quality results for every run.