Pipelines, storage, modelling and governance that make data reliable for analysis and AI.
The shape of a working pipeline — sources, landing, transformation, serving — and the decisions at each boundary.
When data should move in batches, when it should stream, and the real cost of "real time".
Facts, dimensions and the grain — the vocabulary that keeps a warehouse usable past its first year.
Tests, expectations and freshness checks — plus how to raise an alarm people actually read.
Turning a pile of jobs into a graph that runs itself and tells you when it does not.
Pagination, rate limits, auth, awkward CSVs and the file that arrives twice — the unglamorous craft of getting data in.
Agreeing what a producer owes a consumer, and evolving it without a 3am surprise.
Partitioning, clustering, file sizes and the handful of habits that cut a warehouse bill by most of it.
The data engineering behind a model: point-in-time correctness, training/serving consistency and backfills.
Keeping a copy of an operational database in the warehouse without hammering it or missing deletes.
25 in this topic
What data engineers do, how data flows from source systems to analysis and AI, and the core skills involved.
Data engineering 2 min read 28 Jan 2026 For people
The difference between transforming data before loading and after, and why modern warehouses shifted the default to ELT.
Data engineering 2 min read 27 Jan 2026 For people
The three main architectures for analytical data storage, what each is good for and how they are converging.
Data engineering 1 min read 26 Jan 2026 For people
Star schemas, facts and dimensions: how to design tables that make analysis fast, consistent and easy to understand.
Data engineering 2 min read 25 Jan 2026 For people
When to process data in scheduled batches and when real-time streaming is worth the added complexity.
Data engineering 1 min read 24 Jan 2026 For people
How orchestration tools schedule and manage data pipelines, handle dependencies and failures, and what to look for.
Data engineering 2 min read 23 Jan 2026 For people
Testing data pipelines like software: unit tests, schema checks, data quality assertions and freshness monitoring.
Data engineering 2 min read 22 Jan 2026 For people
How change data capture streams inserts, updates and deletes from databases, and why it beats repeated full extracts.
Data engineering 2 min read 21 Jan 2026 For people
Keeping production data trustworthy with automated checks, freshness alerts, anomaly detection and clear ownership.
Data engineering 2 min read 20 Jan 2026 For people
Agreements between data producers and consumers about schema, meaning and quality that stop pipelines breaking.
Data engineering 2 min read 19 Jan 2026 For people
Ownership, definitions, access control, quality and lifecycle: the building blocks of governing data without slowing everyone down.
Data engineering 2 min read 18 Jan 2026 For people
What Spark is, how it processes large datasets across many machines, and when you need it — or don't.
Data engineering 2 min read 17 Jan 2026 For people
How analytics engineering brings software practices to SQL transformations, and how dbt models, tests and documents data.
Data engineering 2 min read 16 Jan 2026 For people
The building blocks of cloud data platforms — storage, compute, warehouses, orchestration and governance — and how to choose.
Data engineering 2 min read 15 Jan 2026 For people
The decentralised approach to data architecture: domain ownership, data as a product, self-service platforms and federated governance.
Data engineering 2 min read 14 Jan 2026 For people
Common patterns for building reliable pipelines: idempotency, incremental loads, backfills and layered architectures.
Data engineering 1 min read 13 Jan 2026 For people
How to track changes to descriptive attributes over time in data warehouses, with SCD types explained.
Data engineering 1 min read 12 Jan 2026 For people
An introduction to Kafka's concepts — topics, partitions, producers and consumers — and when event streaming fits.
Data engineering 1 min read 11 Jan 2026 For people
Making analytical queries faster and cheaper: execution plans, indexes, partitioning and query patterns.
Data engineering 1 min read 10 Jan 2026 For people
The pipelines behind RAG and LLM features: ingestion, parsing, embedding, indexing and freshness.
Data engineering 1 min read 9 Jan 2026 For people