Streaming vs Batch Processing Architectures: A Data Engineer’s Guide

Every data platform eventually confronts a single design question that shapes its cost, its latency, and its operational burden: should a given computation run over a finite dataset on a schedule, or continuously over an endless flow of events? Batch processing computes over bounded data in periodic runs, while stream processing computes incrementally over an unbounded sequence of events as they arrive. The two paradigms are often presented as rivals, but they are better understood as points on a single continuum that trades data freshness against cost, throughput, and correctness machinery. This guide provides a mechanistic, vendor-neutral model of that continuum for engineers who already operate pipelines and now need a rigorous basis for deciding where each new workload belongs.

The discussion covers the batch-versus-streaming trade-off surface, the Lambda and Kappa architecture debate, the execution split between micro-batch and event-at-a-time engines, the correctness semantics that make streaming trustworthy (event time, watermarks, windowing, and exactly-once processing), an honest account of where batch remains the correct choice, and the convergence of the two paradigms observed across 2026 toward unified engines and the streaming lakehouse.

Two Ways to Compute Over Data

The distinction between batch and streaming begins with the shape of the input. A bounded dataset is finite and complete: yesterday’s transactions, a snapshot of a table, a directory of log files. An unbounded dataset is an endless sequence of events that has no defined end, such as clickstream events, sensor readings, or a change-data-capture feed from a production database. Batch processing operates over bounded data in scheduled runs, producing a result after it has observed the entire input. Stream processing operates over unbounded data continuously, updating results incrementally as each event arrives.

This difference in input shape propagates into every operational property of a pipeline. The most useful way to reason about the choice is as a trade-off surface with three axes: latency, throughput, and cost. Latency is the delay between an event occurring and its effect appearing in a result. Throughput is the volume of records the system processes per unit of time. Cost is the compute and operational expense of running the system. These three cannot be optimized independently. Lowering latency toward the millisecond range generally requires always-on compute and additional correctness machinery, which raises both cost and operational complexity. Tolerating higher latency allows work to be amortized into scheduled bursts, which lowers steady-state cost and simplifies recovery.

A central insight, formalized by the Google Dataflow model (Akidau et al., The Dataflow Model, PVLDB 2015), dissolves the apparent opposition between the two paradigms. In that framework, a batch job is simply a stream computation over a bounded input assigned to a single global window that closes when the input is exhausted. Streaming generalizes batch rather than replacing it: the same logical operations (filtering, joining, aggregating) apply in both cases, and the differences reduce to when results are emitted and how completeness is judged. This reframing is more than a rhetorical convenience, because it underpins the unified engines discussed later, in which one API expresses both bounded and unbounded computation.

The following table summarizes how the paradigm choice affects the operational properties an engineer must plan around.

Lambda and Kappa: The Architecture Debate

Once an organization needs both historical accuracy and low-latency views, it confronts an architectural question that predates modern unified engines. The two canonical answers are the Lambda architecture and the Kappa architecture, and understanding their motivations clarifies design decisions that remain relevant even where neither is adopted by name.

The Lambda Architecture

The Lambda architecture was described by Nathan Marz, the creator of Apache Storm, around 2011 and later formalized with James Warren in the book Big Data (Manning, 2015). It composes three layers. The batch layer holds an immutable master dataset and precomputes comprehensive batch views over all historical data, prioritizing accuracy and completeness. The speed layer processes only recent data with low latency, producing approximate or incremental real-time views that compensate for the batch layer’s delay. The serving layer indexes and merges the outputs of both layers so that a query sees a combined result: authoritative history from the batch layer plus the most recent events from the speed layer.

The Lambda architecture achieves both accuracy and freshness, but at a well-known cost: the same business logic must be implemented twice, once in a batch engine and once in a stream engine, and the two implementations must be kept semantically equivalent. Any divergence between them produces inconsistent results at the serving layer, and every change to the computation must be applied and validated in both places. This dual-codebase reconciliation burden is the defining pain point of the pattern.

The Kappa Architecture

Jay Kreps, a co-creator of Apache Kafka, proposed an alternative in the 2014 essay Questioning the Lambda Architecture (O’Reilly Radar). The argument is direct: if a stream processor is sufficiently expressive and reliable, the separate batch layer is redundant. The Kappa architecture keeps a single streaming layer and backs it with a durable, replayable log, typically Kafka, that retains the raw event history. There are no longer two code paths. When the computation logic changes, or a bug is fixed, “reprocessing” means starting a new instance of the streaming job from the beginning of the retained log and letting it recompute the output, then switching consumers to the new result. The nightly recompute of the batch layer is replaced by a replay of the same code that serves live traffic.

The Kappa architecture depends on two properties of its log. The log must be durable and retained long enough to replay whatever history a recomputation requires, and it must preserve ordering per partition so that replay is deterministic. Kafka provides both, which is why it sits at the center of most Kappa deployments. Implementing a reliable consumer over that log is itself a substantial task; the mechanics of offset management, consumer groups, and rebalancing are treated in this guide to implementing a Kafka consumer in Python. The events feeding such a log frequently originate from operational databases through change-data-capture, in which row-level changes are streamed as events; the change-data-capture pattern with Debezium and Kafka is a canonical source that makes Kappa practical for database-derived data.

The industry has drifted toward Kappa because a single codebase is cheaper to maintain and because modern stream engines are expressive enough to carry the full computation. Lambda has not disappeared, however. It survives in domains where an independent, authoritative batch recomputation over an immutable master dataset serves as a correctness audit and a reconciliation baseline, particularly where regulatory or financial reporting demands a reproducible ground truth computed separately from the live path. In practice, many teams run a hybrid: a Kappa-style streaming path for freshness and a periodic batch job that reconciles and corrects, which is closer to Lambda in spirit than either label admits.

Execution Models: Micro-Batch and Event-at-a-Time

Below the architectural layer sits a second decision that determines a pipeline’s latency floor and recovery behavior: how the engine physically executes the stream. Two designs dominate, and they differ in the unit of work they process at a time.

Micro-batch execution

Spark Structured Streaming uses micro-batch execution by default. The incoming stream is divided into a sequence of small, deterministic batches, and each batch is executed as an ordinary Spark job over the records that accumulated during a short interval. This design inherits the fault-tolerance and exactly-once guarantees of Spark’s batch engine almost for free, because recovery means re-running a deterministic batch. According to the Spark Structured Streaming Programming Guide (version 4.1.x, 2026), the default micro-batch engine achieves end-to-end latencies as low as roughly 100 milliseconds with exactly-once guarantees. The trade-off is that the batch boundary imposes a latency floor: a result cannot appear until its micro-batch has been formed and executed. Micro-batch execution is well suited to high-throughput workloads where a latency of a fraction of a second is acceptable.

Spark has offered lower-latency options over time. A Continuous Processing mode, introduced experimentally in Spark 2.3, processes records with latency near one millisecond but provides only at-least-once guarantees. More recently, Spark 4.1 added a Real-Time Mode for Structured Streaming that targets sub-second and single-digit-millisecond latency for stateless tasks (Spark 4.1.0 release notes and the associated Databricks engineering blog, 2026).

Event-at-a-time execution

Apache Flink processes each record as it arrives, one event at a time, through a long-lived graph of operators. There is no batch boundary to wait for, so latency is bounded by the processing of an individual record rather than by an interval. This event-at-a-time model gives Flink native event-time semantics and a rich model of state and timers, which makes it a natural fit for complex, low-latency stream computations. Pattern-matching over event streams is one such application; the complex event processing pipeline with Flink CEP illustrates how the event-at-a-time model supports detecting temporal patterns across a continuous stream.

Apache Beam occupies a distinct position in this landscape. Rather than being an execution engine, Beam is a programming model that implements the Dataflow abstraction and compiles the same pipeline to different execution engines, called runners, including Flink and Spark. The Beam release line reached version 2.74.0 (2026-06-02), which added a Spark 4 runner for the Java software development kit (Beam downloads page, 2026). Beam lets an engineer write a computation once against the Dataflow model and choose the execution backend separately, which is the clearest embodiment of the “batch is a special case of streaming” principle.

The engine version landscape as of mid-2026 is worth noting for teams selecting a platform. Apache Flink reached 2.3.0 (2026-06-25), with a long-term-support line at 1.20.x, following the 2.0.0 major release in March 2025. Apache Spark’s current stable releases are 4.1.2 (2026-05-21) and 4.0.3 on the 4.0 line. Apache Kafka reached 4.3.1 (2026-06-25); notably, Kafka 4.0.0 removed ZooKeeper entirely, making the KRaft consensus protocol the only operating mode (Flink, Spark, and Kafka download and release pages, 2026).

Getting Streaming Correct: Time, Watermarks, Windows, and Semantics

The difficulty of streaming is not moving events quickly; it is producing correct answers over data that arrives late, out of order, and without a natural end. This section covers the machinery that makes streaming results trustworthy, which is the substance that separates a robust pipeline from one that silently produces wrong aggregates.

Event time versus processing time

Event time is the moment an event actually occurred, recorded as a timestamp embedded in the record itself. Processing time is the moment the pipeline observes the event, which depends on network delay, buffering, and system load. In any real distributed system these two clocks diverge, and the gap between them, called event-time skew, is variable. A mobile device that loses connectivity may deliver an event minutes after it occurred. Because business questions are almost always framed in terms of when events happened rather than when a server saw them, correct windowed aggregation must be computed in event time. Aggregating in processing time is simpler but produces results that shift depending on system latency, which makes them non-reproducible.

Watermarks

A watermark is a monotonic assertion about completeness in the event-time domain. A watermark of value T declares that the system expects no further events with a timestamp at or before T. This is precisely the mechanism a streaming system uses to decide when an event-time window is complete enough to close and emit a result. A watermark that advances aggressively produces low-latency results but risks excluding genuinely late events; a conservative watermark waits longer, includes more late data, and increases latency. The watermark is therefore the tuning knob that trades completeness against latency, and it is defined per pipeline based on how late data is expected to arrive.

Events that arrive after the watermark has passed their window are late data. Systems handle late data through configurable policies: an allowed-lateness period keeps a window’s state alive for a grace interval, triggers can fire refined results as more data arrives, and retractions can withdraw and correct a previously emitted result. These options are drawn directly from the Dataflow model’s framing of when results are materialized and how refinements relate to earlier outputs.

Windowing

Windowing divides an unbounded stream into finite chunks over which aggregation is defined. Three window types are canonical. A tumbling window is fixed-size, contiguous, and non-overlapping, so each event belongs to exactly one window; a five-minute tumbling window partitions time into adjacent five-minute blocks. A sliding window has a fixed size and a separate slide interval and therefore overlaps, so a single event can belong to several windows; a ten-minute window that advances every minute is a sliding window. A session window is data-driven and gap-based: it groups events separated by less than a configured inactivity gap and closes after the gap elapses, producing variable-length windows that are not aligned across keys, which is well suited to modeling bursts of user activity.

Delivery semantics and exactly-once

Delivery semantics describe how many times an event’s effect can be reflected in the output when failures and retries occur. At-most-once processing may drop records and never duplicates them; it is fire-and-forget and cheapest. At-least-once processing never loses a record but may apply it more than once after a retry or recovery, producing duplicates. Exactly-once processing guarantees that each record affects the computed state exactly once despite failures, which is what most correctness-sensitive aggregations require.

Flink achieves exactly-once state consistency through asynchronous barrier snapshotting, a variant of the Chandy-Lamport distributed snapshot algorithm (Flink Stateful Stream Processing documentation, 2026). The coordinating JobManager periodically injects special records called checkpoint barriers into the streams at the sources. As a barrier flows downstream, it separates the records that belong to the current snapshot from those that belong to the next one. When an operator has multiple input channels, it performs barrier alignment: it waits until the barrier has arrived on every input channel before taking its snapshot, buffering records that arrive after the barrier on faster channels. Once all operators have snapshotted their state, the checkpoint is complete and can be used to restore the entire job after a failure.

Internal exactly-once state is not sufficient by itself. To make results exactly-once all the way to an external system, the sink must participate in the checkpoint through a two-phase commit: it prepares (writes data in an uncommitted transaction) as part of a checkpoint and commits only after the checkpoint has completed successfully, so that a failure before completion leaves nothing visible to downstream readers. Flink implements this through a transactional committer for sinks that support transactions or idempotent writes. Kafka’s transactional producer and its durable, ordered-per-partition log are what make such end-to-end guarantees possible on the sink side.

Barrier alignment has a cost. Under backpressure, an operator can stall while waiting for a barrier on a slow channel, which delays the checkpoint and raises latency. This motivated unaligned checkpoints, in which barriers overtake buffered records and the in-flight data is included in the snapshot instead, trading a larger checkpoint for shorter alignment delay. The default in Flink remains exactly-once with aligned checkpoints, and the choice to relax it is a deliberate latency optimization.

When Batch Remains the Right Choice, and the Convergence

The correctness machinery above should make one point clear: streaming buys freshness at the price of real complexity. For many workloads that price is not justified, and batch remains the correct and often superior choice. Recognizing these cases is as important as knowing how to build a stream.

Batch is the better fit for large historical backfills, where terabytes of accumulated data must be reprocessed and there is no latency requirement at all. It is preferable for full-refresh reproducibility, where the ability to re-run a deterministic job and obtain an identical, auditable result is worth more than freshness. It suits complex multi-source joins that tolerate latency, cost-sensitive periodic reporting where always-on compute would be wasteful, and the construction of machine-learning training sets, which are inherently snapshots over a bounded, versioned dataset. It also remains the natural home for correctness-audit reprocessing, the independent recomputation that underpins the surviving uses of Lambda.

The batch layer has its own mature tooling. Scheduled batch pipelines are commonly orchestrated with a workflow scheduler; the Apache Airflow orchestration guide describes how directed acyclic graphs express batch dependencies and retries. Transformation logic over warehouse tables is frequently expressed with a batch-first tool as covered in the dbt transformation pipeline guide. Batch outputs typically land in columnar files whose layout is examined in the Parquet and Arrow internals guide, and interactive analysis over such files increasingly runs in in-process engines compared in the DuckDB and Polars comparison.

The 2026 convergence

The sharp dichotomy between batch and streaming is eroding along three lines. The first is unified engines and APIs: Beam’s single model runs on multiple runners, and both Spark and Flink execute bounded and unbounded jobs through largely shared machinery, so the same logic serves both modes. The second is incremental processing: rather than fully reloading a dataset, a pipeline recomputes only the portion that changed, which imports a streaming efficiency into what looks like a batch job. The third is the streaming lakehouse, in which a single table serves both batch queries and streaming reads and writes.

Open table formats provide the foundation for the lakehouse by adding transactional guarantees and metadata to files in object storage; the trade-offs among them are examined in the Iceberg, Delta Lake, and Hudi comparison. A streaming-native format, Apache Paimon, an Apache Top-Level Project since 2024, extends this idea further by combining a log-structured merge-tree with a changelog stream, so that the same table can be written and read as a continuous stream while also supporting batch queries and incremental reads (Apache Paimon project, 2026). The practical effect is that Lambda’s two separate stores collapse into one, which removes the reconciliation burden that motivated the Kappa argument in the first place.

A Decision Checklist

The following questions guide a workload toward batch or streaming without prescribing a specific engine. They are ordered so that the strongest determinants come first.

When several answers point in the same direction, the decision is clear. When they conflict, the convergence tooling offers a middle path: build on a unified engine and a lakehouse table so that a workload can begin as batch and gain a streaming read later without a rewrite, deferring the commitment until the freshness requirement is genuinely established.

Frequently Asked Questions

Is streaming always more expensive than batch?

Not in a fixed ratio. Streaming runs always-on compute, which produces a steady cost that pays for freshness, while batch runs in scheduled bursts whose cost is amortized. Whether streaming costs more for a given workload depends on the state size, the shuffle behavior, and how continuously the data actually arrives. It is more accurate to reason about the shape of the trade-off, always-on versus bursty, than to apply a single multiplier.

Why not just use processing time instead of event time?

Processing-time aggregation is simpler because it ignores when events actually occurred, but its results shift with system latency and are therefore non-reproducible. If a network delay causes events to arrive late, a processing-time window attributes them to the wrong interval. Business questions are almost always framed in event time, so correct windowed results require it, which in turn requires watermarks to judge completeness.

What is the practical difference between at-least-once and exactly-once?

At-least-once never loses a record but may apply it more than once after a failure, producing duplicates. Exactly-once guarantees each record affects the result exactly once, at the cost of checkpointing and, for external sinks, a two-phase commit. If the downstream system deduplicates or the operation is idempotent, at-least-once can deliver equivalent correctness more cheaply; exactly-once is warranted when duplicates genuinely corrupt the output.

Does the convergence toward unified engines make Lambda and Kappa obsolete?

It reduces the cost that motivated the debate rather than settling it. Streaming lakehouse table formats collapse Lambda’s two stores into one, which removes much of the reconciliation burden. An independent batch recomputation still has value as a correctness audit in regulated or reconciliation-heavy domains, so Lambda-style patterns persist even as the tooling makes a single code path easier to maintain.

Is micro-batch a form of batch or streaming?

It is streaming implemented by repeatedly running very small batch jobs. Spark Structured Streaming chops an unbounded stream into short, deterministic micro-batches, which inherits batch-style recovery and exactly-once guarantees while continuously producing results. It sits between the two paradigms and illustrates why the batch-streaming boundary is better viewed as a continuum than a hard line.

References

• Akidau, T., et al. The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing. PVLDB 8(12), 2015.
• Kreps, J. Questioning the Lambda Architecture. O’Reilly Radar, 2014.
• Apache Flink. Stateful Stream Processing: Checkpoints, Barriers, and Exactly-Once. Apache Software Foundation, 2026.
• Apache Spark. Structured Streaming Programming Guide. Apache Software Foundation, 2026.
• Apache Beam. Beam Programming Guide and Downloads. Apache Software Foundation, 2026.
• Apache Paimon. Apache Paimon: Streaming Lakehouse Table Format. Apache Software Foundation, 2026.

Conclusion

Batch and stream processing are not opposing technologies but positions on a single trade-off surface defined by latency, throughput, and cost. The Dataflow model makes this concrete by treating a batch job as a stream computation over a bounded input in a single global window, which is why the same engines and APIs increasingly express both. The architectural debate between Lambda and Kappa reduces to whether an independent batch recomputation is worth its dual-codebase burden, and the industry has drifted toward the single-path Kappa answer while retaining Lambda where reconciliation demands it. Beneath the architecture, the choice of execution model, micro-batch or event-at-a-time, sets a pipeline’s latency floor and recovery behavior, and the correctness machinery of event time, watermarks, windowing, and exactly-once processing is what makes streaming results trustworthy rather than merely fast.

The most durable guidance is to resist treating either paradigm as a default. A workload should be placed on the continuum according to how fresh its result must be and what that freshness is worth, and the 2026 convergence toward unified engines and streaming lakehouse tables increasingly allows that decision to be deferred and revised without a rewrite. An engineer who understands the trade-off surface, rather than memorizing a preferred stack, is equipped to make that judgment correctly for each new dataset.