4.2. Stream Processing

Stream processing computes over data that never ends, which breaks the batch assumption that you can wait for all the input. The core difficulty is time itself: events arrive out of order and late, so the system must distinguish when something happened (event time) from when it was seen (processing time), and decide — via watermarks and windows — how long to wait for stragglers before emitting a result. This chapter builds the Dataflow model that made these questions answerable and grounds it in Apache Flink’s stateful engine.

Topics Covered

• 4.2.1. Bounded vs. Unbounded Data: Distinguishes finite bounded datasets from infinite unbounded streams and why the latter breaks batch assumptions.
• 4.2.2. Event Time vs. Processing Time: The Critical Distinction: Establishes the critical distinction between when an event happened and when the system observed it.
• 4.2.3. Windowing: Tumbling, Sliding, and Session Windows: Covers tumbling, sliding, and session windows for bounding computation over an unbounded stream.
• 4.2.4. Watermarks: The Art of Managing Late Data: Explains watermarks, the mechanism that decides how long to wait for late data before emitting a result.
• 4.2.5. Triggers and Accumulation Modes: Covers triggers that control when results fire and accumulation modes that control how they update.
• 4.2.6. Apache Flink: Depth of Stateful Stream Processing: Grounds the streaming model in Flink’s stateful engine, checkpointing, and exactly-once processing.