Broken pipelines produce wrong numbers silently. Testing catches problems before they reach dashboards and models.
Test the Code
Unit tests check transformation logic on small, hand-made inputs: does the deduplication keep the latest record? Does currency conversion handle missing rates?
Test the Data
Data can be wrong even when code is correct. Add assertions that run with every pipeline:
- Schema: expected columns and types exist.
- Uniqueness: primary keys aren't duplicated.
- Not null: required fields are populated.
- Accepted values: codes come from the allowed list.
- Relationships: every order's customer exists in the customer table.
- Ranges: amounts are positive, dates are plausible.
- Volume: row counts within expected bounds compared with previous runs.
Tools such as dbt tests and Great Expectations make these easy to define.
Test Freshness
Check that data arrived when expected. Stale data displayed as current is a common, damaging failure.
Where to Test
- At ingestion: catch bad source data early.
- After transformations: check business logic.
- Before publishing: gate what reaches users.
Decide What Happens on Failure
Some failures should stop the pipeline; others should warn. Define severity per test.
Integration Testing
Run the pipeline end to end on a representative sample or staging environment before deploying changes.
Treat Failures as Signals
Investigate root causes, fix them at the source and add a test so the same problem is caught next time.