Most modern data platforms run in the cloud, assembled from managed services.
The Building Blocks
- Object storage: cheap, durable storage for raw and processed files (the basis of data lakes).
- Data warehouse or lakehouse engine: scalable SQL analytics.
- Ingestion tools: connectors and CDC services that bring in data from applications and databases.
- Processing: SQL transformations, Spark and serverless functions.
- Orchestration: scheduling and dependency management.
- Streaming: managed event platforms for real-time data.
- Governance: catalogues, lineage, access control and data masking.
- BI and machine learning services on top.
Major Options
The large cloud providers offer full suites, and independent platforms provide warehouses and lakehouses that run across clouds. Many organisations combine managed services with open-source tools.
Choosing
- Fit with your existing cloud provider, skills and systems.
- Workloads: reporting, machine learning, streaming, data sharing.
- Cost model: storage, compute billed per second or per query, egress between regions and clouds.
- Security, compliance and data residency requirements.
- Openness: open table formats and standard SQL reduce lock-in.
Managing Cost
Separate storage from compute, pause idle compute, partition and cluster large tables, monitor expensive queries, and set budgets and alerts.
Start Small
Begin with the few sources and use cases that deliver clear value, and design for growth rather than building everything up front.