Stream Processing Patterns Every Data Engineer Should Know

March 15, 2026  ·  12 min read

Stream processing has a learning curve that batch processing doesn't, and most of that curve is about patterns — specific approaches to solving common problems in a streaming context that don't translate directly from batch thinking.

Here are the eight patterns I reach for most often in production pipelines, with honest notes on where each one will get you into trouble.

1. Tumbling Windows

A tumbling window divides time into fixed, non-overlapping chunks. All events in a 5-minute window are aggregated together; the window closes, results are emitted, and the next window starts. No event belongs to more than one window.

This is the simplest windowing pattern and the right choice when you want discrete time periods with no overlap — hourly counts, daily summaries, per-minute throughput metrics. The limitation: an event that arrives at 11:59:59 and an event that arrives at 12:00:01 are in completely different windows, even though they're two seconds apart. For most use cases that's fine. For use cases that care about edge transitions, it isn't.

2. Sliding Windows

A sliding window has a fixed duration that moves forward in time continuously. A "last 5 minutes" view is a sliding window — as time passes, the window slides forward, old events drop off, new ones enter. Each event can belong to multiple windows.

This is what you want for "current state" dashboards: error rate over the last 15 minutes, active sessions in the last hour, revenue in the last 30 minutes. The tradeoff is state size — you need to retain all events for the full window duration. A sliding 24-hour window on high-volume event streams requires holding 24 hours of events in memory or efficiently in a state store.

3. Session Windows

Session windows are defined by activity gaps rather than fixed time periods. A session starts with the first event from a user, extends as long as events keep arriving within a timeout period (say, 30 minutes of inactivity), and closes when the timeout expires. The duration is variable and data-driven.

This is the natural fit for user session analytics, where "session" means a coherent period of activity. The complexity: you can't know when a session ends until the timeout fires, which means results are delayed by the timeout duration. And session state can be large — 100,000 concurrent users means 100,000 active session states.

4. Stream-Table Joins

Enriching a stream of events with reference data from a slowly-changing table is one of the most common patterns in production pipelines. An order event gets enriched with customer tier data. A click event gets enriched with campaign metadata. The pattern requires one side to be treated as a lookup table rather than a stream.

The implementation choice matters: do you cache the table in memory (fast but stale), load from an external store on each event (slow but fresh), or replicate the table as a changelog topic that updates the local cache in real time (complex but correct)? For tables that change frequently, the changelog approach is usually the right answer.

5. Exactly-Once Processing

Getting this right is harder than it sounds, and many pipelines that claim exactly-once are actually at-least-once with deduplication bolted on. True exactly-once requires transactional coordination between the source offset commits and the destination writes — so that a failure and retry doesn't produce duplicate results.

The practical advice: use at-least-once delivery combined with idempotent writes at the destination wherever possible. It's simpler, more portable, and handles most cases. Reserve exactly-once semantics for financial calculations or any use case where duplicates have real-world consequences.

6. Late Event Handling

Mobile clients, offline-capable apps, and slow network hops all produce events that arrive late — sometimes minutes late, sometimes hours. If your pipeline uses event time for windowing, late arrivals need special handling or they'll either be silently dropped or corrupt closed window results.

The standard patterns are: accept late events up to a configurable watermark delay, route events beyond the watermark to a side output for separate handling, or use a re-emitting window that corrects results when late data arrives. The right choice depends on whether your downstream consumers can handle result corrections.

7. Dead Letter Queues

Any event that fails processing — schema validation error, deserialization failure, missing required field — needs somewhere to go that isn't a crash loop. Dead letter queues (DLQs) are the pattern: failed events route to a side stream that can be monitored, inspected, and replayed once the processing logic is fixed.

DLQs are table stakes for production pipelines. Without them, a single malformed event can halt processing while the pipeline retries forever. With them, the malformed event is isolated and the healthy stream keeps moving. Always include a failure reason and timestamp in the DLQ record.

8. Change Data Capture as a Stream Source

CDC turns database change logs into event streams, making relational data sources available for stream processing without application-level changes. Insert, update, and delete operations become events that your pipeline can consume and react to in real time.

The catches: CDC requires access to the database's replication log, which isn't always available or straightforward to configure. Schema changes in the source table require careful handling — a column rename or type change can break CDC consumers downstream. And initial snapshot loading (getting the existing table state before streaming starts) is a solved but non-trivial problem that needs planning.

Despite the setup complexity, CDC is often the cleanest way to integrate legacy relational systems into a stream processing architecture without instrumenting every application that writes to those systems.