Change data capture (CDC) tracks changes in a source database — inserts, updates and deletes — and delivers them to other systems.
Why CDC
Repeatedly copying whole tables is slow, heavy on the source database and misses deletions. CDC moves only what changed, often in near real time.
Approaches
- Log-based CDC: read the database's transaction log (write-ahead log or binlog). Captures every change, including deletes, with minimal load on the source. The preferred approach.
- Timestamp-based: query rows with an "updated at" value after the last run. Simple, but misses deletes and relies on timestamps being maintained correctly.
- Triggers: database triggers write changes to a separate table. Adds load and complexity to the source.
What CDC Produces
A stream of change events with the operation, the row's data (before and after), and ordering information. Downstream systems apply these to keep copies in sync.
Common Uses
- Keeping a warehouse or lakehouse up to date.
- Feeding search indexes and caches.
- Event-driven integrations between services.
- Auditing changes.
Challenges
- Schema changes in the source must be handled.
- Ordering and exactly-once delivery require care.
- Initial snapshots must be combined with ongoing changes.
- Sensitive data flows onward, so governance applies.
Tools
Open-source connectors such as Debezium, and managed services from cloud and data-integration providers.