Data can be processed in batches at intervals or as a continuous stream as events happen.
Batch Processing
Collect data over a period, then process it together — hourly, nightly or weekly.
Strengths: simpler to build, test and rerun; efficient for large volumes; easier to guarantee completeness. Suits: reporting, billing, model training, most analytics.
Stream Processing
Process each event (or small window of events) within seconds or less.
Strengths: low latency; enables immediate reactions. Suits: fraud detection during payment, live monitoring and alerting, real-time personalisation, IoT sensor processing.
Challenges of Streaming
- Late and out-of-order events need handling with time windows and watermarks.
- Exactly-once processing is hard to guarantee.
- State management for aggregations across time.
- More complex operations, monitoring and debugging.
Common Tools
Event platforms such as Apache Kafka and cloud equivalents carry streams; processing frameworks such as Apache Flink and Spark Structured Streaming transform them.
Choosing
Ask how quickly a decision genuinely needs the data. "Real time" is often requested when hourly would do. Start with batch, and use streaming where latency creates real value.
Hybrid Approaches
Many systems combine them: streaming for immediate alerts and batch for complete, corrected figures later.