Author: kongastral

Every data platform eventually confronts a single design question that shapes its cost, its latency, and its operational burden: should a given computation run over a finite dataset on a schedule, or continuously over an endless flow of events? Batch processing computes over bounded data in periodic runs, while stream processing computes incrementally over an unbounded sequence of events as they arrive. The two paradigms are often presented as rivals, but they are better understood as points on a single continuum that trades data freshness against cost, throughput, and correctness machinery. This guide provides a mechanistic, vendor-neutral model of that continuum for engineers who already operate pipelines and now need a rigorous basis for deciding where each new workload belongs.

The discussion covers the batch-versus-streaming trade-off surface, the Lambda and Kappa architecture debate, the execution split between micro-batch and event-at-a-time engines, the correctness semantics that make streaming trustworthy (event time, watermarks, windowing, and exactly-once processing), an honest account of where batch remains the correct choice, and the convergence of the two paradigms observed across 2026 toward unified engines and the streaming lakehouse.

Two Ways to Compute Over Data

The distinction between batch and streaming begins with the shape of the input. A bounded dataset is finite and complete: yesterday’s transactions, a snapshot of a table, a directory of log files. An unbounded dataset is an endless sequence of events that has no defined end, such as clickstream events, sensor readings, or a change-data-capture feed from a production database. Batch processing operates over bounded data in scheduled runs, producing a result after it has observed the entire input. Stream processing operates over unbounded data continuously, updating results incrementally as each event arrives.

This difference in input shape propagates into every operational property of a pipeline. The most useful way to reason about the choice is as a trade-off surface with three axes: latency, throughput, and cost. Latency is the delay between an event occurring and its effect appearing in a result. Throughput is the volume of records the system processes per unit of time. Cost is the compute and operational expense of running the system. These three cannot be optimized independently. Lowering latency toward the millisecond range generally requires always-on compute and additional correctness machinery, which raises both cost and operational complexity. Tolerating higher latency allows work to be amortized into scheduled bursts, which lowers steady-state cost and simplifies recovery.

A central insight, formalized by the Google Dataflow model (Akidau et al., The Dataflow Model, PVLDB 2015), dissolves the apparent opposition between the two paradigms. In that framework, a batch job is simply a stream computation over a bounded input assigned to a single global window that closes when the input is exhausted. Streaming generalizes batch rather than replacing it: the same logical operations (filtering, joining, aggregating) apply in both cases, and the differences reduce to when results are emitted and how completeness is judged. This reframing is more than a rhetorical convenience, because it underpins the unified engines discussed later, in which one API expresses both bounded and unbounded computation.

The following table summarizes how the paradigm choice affects the operational properties an engineer must plan around.

Lambda and Kappa: The Architecture Debate

Once an organization needs both historical accuracy and low-latency views, it confronts an architectural question that predates modern unified engines. The two canonical answers are the Lambda architecture and the Kappa architecture, and understanding their motivations clarifies design decisions that remain relevant even where neither is adopted by name.

The Lambda Architecture

The Lambda architecture was described by Nathan Marz, the creator of Apache Storm, around 2011 and later formalized with James Warren in the book Big Data (Manning, 2015). It composes three layers. The batch layer holds an immutable master dataset and precomputes comprehensive batch views over all historical data, prioritizing accuracy and completeness. The speed layer processes only recent data with low latency, producing approximate or incremental real-time views that compensate for the batch layer’s delay. The serving layer indexes and merges the outputs of both layers so that a query sees a combined result: authoritative history from the batch layer plus the most recent events from the speed layer.

The Lambda architecture achieves both accuracy and freshness, but at a well-known cost: the same business logic must be implemented twice, once in a batch engine and once in a stream engine, and the two implementations must be kept semantically equivalent. Any divergence between them produces inconsistent results at the serving layer, and every change to the computation must be applied and validated in both places. This dual-codebase reconciliation burden is the defining pain point of the pattern.

The Kappa Architecture

Jay Kreps, a co-creator of Apache Kafka, proposed an alternative in the 2014 essay Questioning the Lambda Architecture (O’Reilly Radar). The argument is direct: if a stream processor is sufficiently expressive and reliable, the separate batch layer is redundant. The Kappa architecture keeps a single streaming layer and backs it with a durable, replayable log, typically Kafka, that retains the raw event history. There are no longer two code paths. When the computation logic changes, or a bug is fixed, “reprocessing” means starting a new instance of the streaming job from the beginning of the retained log and letting it recompute the output, then switching consumers to the new result. The nightly recompute of the batch layer is replaced by a replay of the same code that serves live traffic.

The Kappa architecture depends on two properties of its log. The log must be durable and retained long enough to replay whatever history a recomputation requires, and it must preserve ordering per partition so that replay is deterministic. Kafka provides both, which is why it sits at the center of most Kappa deployments. Implementing a reliable consumer over that log is itself a substantial task; the mechanics of offset management, consumer groups, and rebalancing are treated in this guide to implementing a Kafka consumer in Python. The events feeding such a log frequently originate from operational databases through change-data-capture, in which row-level changes are streamed as events; the change-data-capture pattern with Debezium and Kafka is a canonical source that makes Kappa practical for database-derived data.

The industry has drifted toward Kappa because a single codebase is cheaper to maintain and because modern stream engines are expressive enough to carry the full computation. Lambda has not disappeared, however. It survives in domains where an independent, authoritative batch recomputation over an immutable master dataset serves as a correctness audit and a reconciliation baseline, particularly where regulatory or financial reporting demands a reproducible ground truth computed separately from the live path. In practice, many teams run a hybrid: a Kappa-style streaming path for freshness and a periodic batch job that reconciles and corrects, which is closer to Lambda in spirit than either label admits.

Execution Models: Micro-Batch and Event-at-a-Time

Below the architectural layer sits a second decision that determines a pipeline’s latency floor and recovery behavior: how the engine physically executes the stream. Two designs dominate, and they differ in the unit of work they process at a time.

Micro-batch execution

Spark Structured Streaming uses micro-batch execution by default. The incoming stream is divided into a sequence of small, deterministic batches, and each batch is executed as an ordinary Spark job over the records that accumulated during a short interval. This design inherits the fault-tolerance and exactly-once guarantees of Spark’s batch engine almost for free, because recovery means re-running a deterministic batch. According to the Spark Structured Streaming Programming Guide (version 4.1.x, 2026), the default micro-batch engine achieves end-to-end latencies as low as roughly 100 milliseconds with exactly-once guarantees. The trade-off is that the batch boundary imposes a latency floor: a result cannot appear until its micro-batch has been formed and executed. Micro-batch execution is well suited to high-throughput workloads where a latency of a fraction of a second is acceptable.

Spark has offered lower-latency options over time. A Continuous Processing mode, introduced experimentally in Spark 2.3, processes records with latency near one millisecond but provides only at-least-once guarantees. More recently, Spark 4.1 added a Real-Time Mode for Structured Streaming that targets sub-second and single-digit-millisecond latency for stateless tasks (Spark 4.1.0 release notes and the associated Databricks engineering blog, 2026).

Event-at-a-time execution

Apache Flink processes each record as it arrives, one event at a time, through a long-lived graph of operators. There is no batch boundary to wait for, so latency is bounded by the processing of an individual record rather than by an interval. This event-at-a-time model gives Flink native event-time semantics and a rich model of state and timers, which makes it a natural fit for complex, low-latency stream computations. Pattern-matching over event streams is one such application; the complex event processing pipeline with Flink CEP illustrates how the event-at-a-time model supports detecting temporal patterns across a continuous stream.

Apache Beam occupies a distinct position in this landscape. Rather than being an execution engine, Beam is a programming model that implements the Dataflow abstraction and compiles the same pipeline to different execution engines, called runners, including Flink and Spark. The Beam release line reached version 2.74.0 (2026-06-02), which added a Spark 4 runner for the Java software development kit (Beam downloads page, 2026). Beam lets an engineer write a computation once against the Dataflow model and choose the execution backend separately, which is the clearest embodiment of the “batch is a special case of streaming” principle.

The engine version landscape as of mid-2026 is worth noting for teams selecting a platform. Apache Flink reached 2.3.0 (2026-06-25), with a long-term-support line at 1.20.x, following the 2.0.0 major release in March 2025. Apache Spark’s current stable releases are 4.1.2 (2026-05-21) and 4.0.3 on the 4.0 line. Apache Kafka reached 4.3.1 (2026-06-25); notably, Kafka 4.0.0 removed ZooKeeper entirely, making the KRaft consensus protocol the only operating mode (Flink, Spark, and Kafka download and release pages, 2026).

Getting Streaming Correct: Time, Watermarks, Windows, and Semantics

The difficulty of streaming is not moving events quickly; it is producing correct answers over data that arrives late, out of order, and without a natural end. This section covers the machinery that makes streaming results trustworthy, which is the substance that separates a robust pipeline from one that silently produces wrong aggregates.

Event time versus processing time

Event time is the moment an event actually occurred, recorded as a timestamp embedded in the record itself. Processing time is the moment the pipeline observes the event, which depends on network delay, buffering, and system load. In any real distributed system these two clocks diverge, and the gap between them, called event-time skew, is variable. A mobile device that loses connectivity may deliver an event minutes after it occurred. Because business questions are almost always framed in terms of when events happened rather than when a server saw them, correct windowed aggregation must be computed in event time. Aggregating in processing time is simpler but produces results that shift depending on system latency, which makes them non-reproducible.

Watermarks

A watermark is a monotonic assertion about completeness in the event-time domain. A watermark of value T declares that the system expects no further events with a timestamp at or before T. This is precisely the mechanism a streaming system uses to decide when an event-time window is complete enough to close and emit a result. A watermark that advances aggressively produces low-latency results but risks excluding genuinely late events; a conservative watermark waits longer, includes more late data, and increases latency. The watermark is therefore the tuning knob that trades completeness against latency, and it is defined per pipeline based on how late data is expected to arrive.

Events that arrive after the watermark has passed their window are late data. Systems handle late data through configurable policies: an allowed-lateness period keeps a window’s state alive for a grace interval, triggers can fire refined results as more data arrives, and retractions can withdraw and correct a previously emitted result. These options are drawn directly from the Dataflow model’s framing of when results are materialized and how refinements relate to earlier outputs.

Windowing

Windowing divides an unbounded stream into finite chunks over which aggregation is defined. Three window types are canonical. A tumbling window is fixed-size, contiguous, and non-overlapping, so each event belongs to exactly one window; a five-minute tumbling window partitions time into adjacent five-minute blocks. A sliding window has a fixed size and a separate slide interval and therefore overlaps, so a single event can belong to several windows; a ten-minute window that advances every minute is a sliding window. A session window is data-driven and gap-based: it groups events separated by less than a configured inactivity gap and closes after the gap elapses, producing variable-length windows that are not aligned across keys, which is well suited to modeling bursts of user activity.

Delivery semantics and exactly-once

Delivery semantics describe how many times an event’s effect can be reflected in the output when failures and retries occur. At-most-once processing may drop records and never duplicates them; it is fire-and-forget and cheapest. At-least-once processing never loses a record but may apply it more than once after a retry or recovery, producing duplicates. Exactly-once processing guarantees that each record affects the computed state exactly once despite failures, which is what most correctness-sensitive aggregations require.

Flink achieves exactly-once state consistency through asynchronous barrier snapshotting, a variant of the Chandy-Lamport distributed snapshot algorithm (Flink Stateful Stream Processing documentation, 2026). The coordinating JobManager periodically injects special records called checkpoint barriers into the streams at the sources. As a barrier flows downstream, it separates the records that belong to the current snapshot from those that belong to the next one. When an operator has multiple input channels, it performs barrier alignment: it waits until the barrier has arrived on every input channel before taking its snapshot, buffering records that arrive after the barrier on faster channels. Once all operators have snapshotted their state, the checkpoint is complete and can be used to restore the entire job after a failure.

Internal exactly-once state is not sufficient by itself. To make results exactly-once all the way to an external system, the sink must participate in the checkpoint through a two-phase commit: it prepares (writes data in an uncommitted transaction) as part of a checkpoint and commits only after the checkpoint has completed successfully, so that a failure before completion leaves nothing visible to downstream readers. Flink implements this through a transactional committer for sinks that support transactions or idempotent writes. Kafka’s transactional producer and its durable, ordered-per-partition log are what make such end-to-end guarantees possible on the sink side.

Barrier alignment has a cost. Under backpressure, an operator can stall while waiting for a barrier on a slow channel, which delays the checkpoint and raises latency. This motivated unaligned checkpoints, in which barriers overtake buffered records and the in-flight data is included in the snapshot instead, trading a larger checkpoint for shorter alignment delay. The default in Flink remains exactly-once with aligned checkpoints, and the choice to relax it is a deliberate latency optimization.

When Batch Remains the Right Choice, and the Convergence

The correctness machinery above should make one point clear: streaming buys freshness at the price of real complexity. For many workloads that price is not justified, and batch remains the correct and often superior choice. Recognizing these cases is as important as knowing how to build a stream.

Batch is the better fit for large historical backfills, where terabytes of accumulated data must be reprocessed and there is no latency requirement at all. It is preferable for full-refresh reproducibility, where the ability to re-run a deterministic job and obtain an identical, auditable result is worth more than freshness. It suits complex multi-source joins that tolerate latency, cost-sensitive periodic reporting where always-on compute would be wasteful, and the construction of machine-learning training sets, which are inherently snapshots over a bounded, versioned dataset. It also remains the natural home for correctness-audit reprocessing, the independent recomputation that underpins the surviving uses of Lambda.

The batch layer has its own mature tooling. Scheduled batch pipelines are commonly orchestrated with a workflow scheduler; the Apache Airflow orchestration guide describes how directed acyclic graphs express batch dependencies and retries. Transformation logic over warehouse tables is frequently expressed with a batch-first tool as covered in the dbt transformation pipeline guide. Batch outputs typically land in columnar files whose layout is examined in the Parquet and Arrow internals guide, and interactive analysis over such files increasingly runs in in-process engines compared in the DuckDB and Polars comparison.

The 2026 convergence

The sharp dichotomy between batch and streaming is eroding along three lines. The first is unified engines and APIs: Beam’s single model runs on multiple runners, and both Spark and Flink execute bounded and unbounded jobs through largely shared machinery, so the same logic serves both modes. The second is incremental processing: rather than fully reloading a dataset, a pipeline recomputes only the portion that changed, which imports a streaming efficiency into what looks like a batch job. The third is the streaming lakehouse, in which a single table serves both batch queries and streaming reads and writes.

Open table formats provide the foundation for the lakehouse by adding transactional guarantees and metadata to files in object storage; the trade-offs among them are examined in the Iceberg, Delta Lake, and Hudi comparison. A streaming-native format, Apache Paimon, an Apache Top-Level Project since 2024, extends this idea further by combining a log-structured merge-tree with a changelog stream, so that the same table can be written and read as a continuous stream while also supporting batch queries and incremental reads (Apache Paimon project, 2026). The practical effect is that Lambda’s two separate stores collapse into one, which removes the reconciliation burden that motivated the Kappa argument in the first place.

A Decision Checklist

The following questions guide a workload toward batch or streaming without prescribing a specific engine. They are ordered so that the strongest determinants come first.

When several answers point in the same direction, the decision is clear. When they conflict, the convergence tooling offers a middle path: build on a unified engine and a lakehouse table so that a workload can begin as batch and gain a streaming read later without a rewrite, deferring the commitment until the freshness requirement is genuinely established.

Frequently Asked Questions

Is streaming always more expensive than batch?

Not in a fixed ratio. Streaming runs always-on compute, which produces a steady cost that pays for freshness, while batch runs in scheduled bursts whose cost is amortized. Whether streaming costs more for a given workload depends on the state size, the shuffle behavior, and how continuously the data actually arrives. It is more accurate to reason about the shape of the trade-off, always-on versus bursty, than to apply a single multiplier.

Why not just use processing time instead of event time?

Processing-time aggregation is simpler because it ignores when events actually occurred, but its results shift with system latency and are therefore non-reproducible. If a network delay causes events to arrive late, a processing-time window attributes them to the wrong interval. Business questions are almost always framed in event time, so correct windowed results require it, which in turn requires watermarks to judge completeness.

What is the practical difference between at-least-once and exactly-once?

At-least-once never loses a record but may apply it more than once after a failure, producing duplicates. Exactly-once guarantees each record affects the result exactly once, at the cost of checkpointing and, for external sinks, a two-phase commit. If the downstream system deduplicates or the operation is idempotent, at-least-once can deliver equivalent correctness more cheaply; exactly-once is warranted when duplicates genuinely corrupt the output.

Does the convergence toward unified engines make Lambda and Kappa obsolete?

It reduces the cost that motivated the debate rather than settling it. Streaming lakehouse table formats collapse Lambda’s two stores into one, which removes much of the reconciliation burden. An independent batch recomputation still has value as a correctness audit in regulated or reconciliation-heavy domains, so Lambda-style patterns persist even as the tooling makes a single code path easier to maintain.

Is micro-batch a form of batch or streaming?

It is streaming implemented by repeatedly running very small batch jobs. Spark Structured Streaming chops an unbounded stream into short, deterministic micro-batches, which inherits batch-style recovery and exactly-once guarantees while continuously producing results. It sits between the two paradigms and illustrates why the batch-streaming boundary is better viewed as a continuum than a hard line.

References

• Akidau, T., et al. The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing. PVLDB 8(12), 2015.
• Kreps, J. Questioning the Lambda Architecture. O’Reilly Radar, 2014.
• Apache Flink. Stateful Stream Processing: Checkpoints, Barriers, and Exactly-Once. Apache Software Foundation, 2026.
• Apache Spark. Structured Streaming Programming Guide. Apache Software Foundation, 2026.
• Apache Beam. Beam Programming Guide and Downloads. Apache Software Foundation, 2026.
• Apache Paimon. Apache Paimon: Streaming Lakehouse Table Format. Apache Software Foundation, 2026.

Conclusion

Batch and stream processing are not opposing technologies but positions on a single trade-off surface defined by latency, throughput, and cost. The Dataflow model makes this concrete by treating a batch job as a stream computation over a bounded input in a single global window, which is why the same engines and APIs increasingly express both. The architectural debate between Lambda and Kappa reduces to whether an independent batch recomputation is worth its dual-codebase burden, and the industry has drifted toward the single-path Kappa answer while retaining Lambda where reconciliation demands it. Beneath the architecture, the choice of execution model, micro-batch or event-at-a-time, sets a pipeline’s latency floor and recovery behavior, and the correctness machinery of event time, watermarks, windowing, and exactly-once processing is what makes streaming results trustworthy rather than merely fast.

The most durable guidance is to resist treating either paradigm as a default. A workload should be placed on the continuum according to how fresh its result must be and what that freshness is worth, and the 2026 convergence toward unified engines and streaming lakehouse tables increasingly allows that decision to be deferred and revised without a rewrite. An engineer who understands the trade-off surface, rather than memorizing a preferred stack, is equipped to make that judgment correctly for each new dataset.

Modern data platforms move records across many independent systems: an application emits an event, an ingestion job lands it in a lake, a transformation framework reshapes it, and a dashboard or a machine-learning model consumes the result. Every one of these handoffs is an implicit agreement about the shape and meaning of the data, and every silent violation of that agreement propagates downstream until a report is wrong or a model degrades. A data contract—a versioned, machine-readable agreement between the team that produces a dataset and the teams that consume it, specifying its schema, semantics, service-level guarantees, and quality expectations—turns that implicit agreement into an explicit, testable artifact. This post examines how data contracts and data quality enforcement work together to make pipelines reliable, and why the two ideas are complementary rather than interchangeable.

The distinction that organizes the discussion is between prevention and detection. A data contract prevents bad data from entering a system by asserting expectations at the boundary where data is produced. Data observability, by contrast, detects problems after data has already landed, by monitoring freshness, volume, and schema drift and raising anomaly alerts. A mature platform uses both. The material below defines the contract as an artifact, presents a taxonomy of six data quality dimensions, surveys the enforcement engines that operate at the record, DataFrame, warehouse, and stream layers, and describes the patterns—shift-left validation, continuous-integration gates, pre-ingestion quarantine, circuit breakers, and dead-letter queues—that put a contract into force.

The Data Contract as a First-Class Artifact

A data contract is a formal, versioned agreement between a data producer and its consumers. The producer is the system or team that emits a dataset—an application service publishing events, a database exposing a table, or an ingestion job writing files. The consumers are the downstream systems and teams that read that dataset: analytics models, dashboards, machine-learning features, and other services. Before contracts, the terms of that relationship lived in tribal knowledge and out-of-date documentation, so a producer could rename a column or change a unit of measurement without warning, and consumers discovered the break only when their output failed. A contract makes the terms explicit and machine-readable, so that a change which would violate them can be caught automatically.

Four elements typically appear in a contract. The schema declares the fields, their types, and their nullability. The semantics describe what each field means, including units, allowed value sets, and business definitions, so that a field named revenue is unambiguous about currency and whether it is gross or net. The service-level agreement (SLA) states operational guarantees such as freshness, expected update frequency, and availability. The quality expectations encode testable rules—for example, that a primary key is unique or that a percentage column falls between zero and one hundred. Figure 1 shows how these four elements sit on the boundary between a producer and its consumers.

A Standard for the Contract: ODCS

Writing a contract in an ad-hoc format leaves every team to invent its own structure, so a shared specification is useful. The Open Data Contract Standard (ODCS) is a vendor-neutral YAML format that describes the agreement between a data producer and its consumers. Its current version is v3.1.0 (Bitol / Linux Foundation AI & Data, as of 2026). ODCS originated as PayPal’s internal data contract template, was open-sourced, and was donated to the Linux Foundation; it is now stewarded by Bitol, a Linux Foundation AI & Data incubation project licensed under Apache 2.0 that was formed on 30 November 2023 when the AIDA User Group and LF AI & Data joined forces (bitol.io, as of 2026). Bitol also stewards the Open Data Product Standard (ODPS), which has reached v1.0.0.

An important point about the current landscape is that the ecosystem is consolidating around a single standard. A separate effort, the Data Contract Specification, is being deprecated and is converging on ODCS v3.1.0. Its maintainers advise users to migrate, and tooling support in the Data Contract CLI and Entropy Data is planned only until the end of 2026 (datacontract-specification.com, as of 2026). A team beginning with contracts in 2026 should therefore standardize on ODCS rather than the older specification. A minimal ODCS-style contract illustrates the artifact concretely.

apiVersion: v3.1.0
kind: DataContract
id: orders-contract
name: Orders
version: 1.2.0
status: active
schema:
  - name: orders
    properties:
      - name: order_id
        logicalType: string
        required: true
        unique: true
      - name: customer_id
        logicalType: string
        required: true
      - name: order_total
        logicalType: number
        required: true
        quality:
          - rule: minimum
            mustBeGreaterThanOrEqualTo: 0
      - name: status
        logicalType: string
        required: true
        quality:
          - rule: enum
            values: [active, shipped, cancelled]
slaProperties:
  - property: freshness
    value: 15
    unit: minute

Treating the contract as versioned code has a direct engineering consequence: a change to the contract is a change to a file in version control, subject to review and to automated compatibility checks. This is what makes the shift-left patterns described later in this post possible, and it links data contracts to the broader practice of treating transformation logic as code, as covered in the guide to dbt for building transformation pipelines.

Six Dimensions of Data Quality

Before a contract can assert quality expectations, quality itself must be defined precisely enough to be measured. A widely used approach decomposes quality into distinct dimensions, each answering a different question about the data. The six dimensions below form a compact and practical taxonomy. Each is defined on first use, because the terms are often used loosely in practice.

Completeness asks whether the expected data is present, with no missing values or absent records where they are required. A completeness check might assert that customer_id is never null, or that a daily table’s row count falls within an expected band. Uniqueness asks whether each real-world entity appears once, with no unintended duplicates; a typical check enforces primary-key uniqueness so that no order_id is repeated. Validity asks whether values conform to a defined format, type, range, or allowed set—the domain of the field. Examples include requiring that status belongs to a fixed set of allowed values, that an email address matches a regular expression, or that an age is not negative.

Accuracy asks whether values correctly describe the real-world entity they represent. This is among the hardest dimensions to test in isolation, because it requires comparison against a trusted system of record rather than an internal rule; a value can be valid and unique yet still wrong. Consistency asks whether values agree across systems, tables, and over time, with no contradictions—for instance, that the sum of an order’s line items equals its recorded total, or that a customer’s country is identical in every table. Timeliness asks whether data is fresh and available within its expected latency or SLA window; a timeliness check verifies that the maximum event timestamp is within the last few minutes or that a partition landed on schedule.

Accuracy and consistency are the two dimensions that resist mechanical enforcement most strongly, because they depend on external reference points and cross-system reconciliation rather than a self-contained rule. A contract can assert them, but verifying them often requires a comparison the pipeline cannot perform on its own. A seventh dimension, integrity—referential integrity, meaning that foreign keys resolve to existing rows—is sometimes added; it can be treated as a specialized form of consistency across tables.

Where Enforcement Happens: Four Layers and Their Engines

A single contract is enforced at several points, because data changes form as it moves. It arrives as individual records at an application boundary, is assembled into in-memory tables (DataFrames) for processing, is materialized as warehouse tables, and travels as messages on streams. Each form has a matching class of validation engine. The essential distinction—already stated for the contract itself—applies here too: the specification declares what must hold, and each engine enforces it in the representation it understands.

Record and DataFrame Boundaries

At the application boundary, data arrives one record at a time—an incoming API request, a single event to be published. Pydantic validates and parses data at this record level. Its current version is v2.13.4 (pydantic/pydantic releases, as of 2026). The validation engine lives in a component called pydantic-core, written in Rust through the pyo3 binding layer, which delivers roughly a fivefold to fiftyfold performance improvement over version 1 depending on the workload (pydantic.dev, as of 2026). Pydantic excels at fast per-record validation at ingestion and API edges, but it operates one object at a time and is not designed for table-scale statistical checks.

Once records are assembled into a DataFrame—an in-memory tabular structure—the relevant unit of validation becomes the column and the table rather than the single object. Pandera provides schema-based, statistical validation for DataFrames. Its current version is 0.32.1 (unionai-oss/pandera releases, as of mid-2026). Pandera 0.19.0 added Polars validation, and by version 0.29 (January 2026) a single schema definition could validate data across pandas, Polars, Dask, Modin, PySpark, and Ibis; a Narwhals-powered backend introduced in 0.32.0 adds lazy validation across Polars, Ibis, and PySpark SQL. A Pandera schema expresses column-level expectations directly.

import pandera as pa
from pandera import Column, Check

schema = pa.DataFrameSchema({
    "order_id":    Column(str, unique=True, nullable=False),
    "customer_id": Column(str, nullable=False),
    "order_total": Column(float, Check.ge(0)),
    "status":      Column(str, Check.isin(["active", "shipped", "cancelled"])),
})

# Raises SchemaError on the first violation, or collects all
# violations when lazy=True.
validated = schema.validate(df, lazy=True)

Warehouse Tables

Inside the warehouse, checks run against materialized tables in SQL. Three engines are common here. dbt tests co-locate assertions with the transformation models that produce a table. dbt Core’s current version is 1.11.12 (dbt-core GitHub releases, as of 2026-07-01), with a Rust-based v2.0.0-alpha in development as the foundation for the dbt Fusion engine. dbt distinguishes data tests—generic and singular assertions that run against materialized data—from native unit tests, which validate SQL logic and were introduced in v1.8; in v1.9 and later, dbt test --resource-type test runs data tests while excluding unit tests. Because dbt tests only cover what dbt materializes, they do not guard the ingestion boundary or streams. The role of dbt in the transformation layer is treated fully in the guide to dbt transformation pipelines.

Great Expectations (GX Core) is a dedicated validation framework built around a reusable library of expressive assertions called Expectations, together with automatic profiling and human-readable validation reports called Data Docs. Its current version is 1.18.2 (Great Expectations changelog, as of 2026-06-26); GX Core 1.0 was a major API redesign that separated the open-source GX Core from the commercial GX Cloud. It supports SQL, Spark, and pandas backends. Its strength is the breadth and reusability of its Expectation library; historically its weakness has been a heavier setup and a steeper learning curve. Figure 5 traces its validation flow.

Soda Core is a second dedicated framework, known for concise checks and a continuous-integration-friendly command-line interface. Its current version is 4.16.0, released 29 June 2026, and it requires Python 3.10 or later (Soda Core release notes, as of 2026-06-29). A notable change is that Soda Core version 4 makes data contracts the default way to define quality rules—a breaking change that moves away from the older SodaCL “checks” syntax toward a contract-based syntax. A concise Soda check reads close to natural language.

checks for orders:
  - row_count > 0
  - missing_count(customer_id) = 0
  - duplicate_count(order_id) = 0
  - invalid_count(status) = 0:
      valid values: [active, shipped, cancelled]
  - freshness(order_time) < 15m

Streams

On a Kafka stream, the contract is the message schema, and the enforcement point is a schema registry—a service that stores schemas and rejects producers whose schema is incompatible with the registered version. Confluent Schema Registry supports Avro, Protobuf, and JSON Schema (referred to as JSON_SR) out of the box on both Confluent Platform and Confluent Cloud (docs.confluent.io, as of 2026). It governs schema evolution through compatibility types: BACKWARD (the default), BACKWARD_TRANSITIVE, FORWARD, FORWARD_TRANSITIVE, FULL, FULL_TRANSITIVE, and NONE. BACKWARD is the default because it allows consumers using the new schema to read data written with the previous schema, which lets a consumer rewind to the start of a topic. Confluent recommends BACKWARD_TRANSITIVE for Protobuf, because adding new message types is not forward compatible, whereas Avro was designed with schema evolution in mind (docs.confluent.io, as of 2026). A schema registry enforces the shape and evolution of messages, not their semantic quality; it will not catch a null in a field that is technically nullable or a value out of an expected range. Streaming schema enforcement pairs naturally with change-data-capture pipelines, as discussed in the guide to change data capture with Debezium and Kafka, and with the consumer-side handling covered in the Kafka consumer implementation guide.

The following table summarizes what each engine does best and where its boundary lies. Every version cited carries its source and date in the prose above.

Enforcement Patterns: Shift-Left, Gates, and Dead-Letter Queues

Having an engine is not the same as having a strategy for where and when it runs. Several established patterns place enforcement at different points in the data’s journey, trading availability against correctness in different ways.

Shift-Left at the Producer Boundary

Shift-left is the practice of moving validation as early as possible—toward the point where data is produced—rather than catching problems late in the pipeline. The term borrows from software testing, where moving tests earlier (“to the left” on a left-to-right timeline) reduces the cost of fixing defects. Applied to data, shift-left means validating at the application or ingestion boundary so that non-conforming data never enters the warehouse at all. This is the cheapest place to catch an error, because the cost of a defect rises as it travels downstream: a bad value caught at the producer affects nothing, whereas the same value caught at the BI or ML layer may have already corrupted reports, retrained a model, or been copied into many derived tables. Figure 3 depicts this rising cost.

Continuous-Integration Gates on Contract Changes

Because a contract is a file under version control, a proposed change to a producer’s schema arrives as a pull request. A continuous-integration (CI) gate is an automated check that runs on that pull request and blocks the merge if the change would break the contract. Two checks matter here: validating that the contract file itself is well formed, and running a backward-compatibility check that determines whether existing consumers can still read data produced under the new schema. This mirrors exactly what a schema registry does at runtime for streams, moved earlier to the moment of code review. A breaking change—removing a required field, narrowing a type—fails the gate and cannot merge until the producer and consumers agree on a migration. Figure 6 shows the contract advancing through versions with a compatibility check at each transition.

Pre-Ingestion Gates and Circuit Breakers

When validation cannot happen strictly at the producer—because the producer is a third party, or the data arrives as files—a pre-ingestion gate places incoming data in a staging area first and requires a validation step to pass before the data is promoted to a production table. The gate fails closed: if the check does not pass, the data does not advance. This quarantines suspect data rather than exposing it to consumers.

A related pattern is the circuit breaker, which halts a running pipeline when a contract check fails beyond a chosen threshold, rather than allowing bad data to propagate. The name is borrowed from electrical engineering, where a breaker interrupts a circuit to prevent damage. In a data pipeline hosted by an orchestrator, a failed check fails the orchestrator task and stops downstream jobs from running. This deliberately trades availability for correctness: a stalled pipeline is often preferable to a fast one that delivers wrong answers, because a visible delay prompts investigation whereas a silent error can persist unnoticed for days. Orchestrators such as those covered in the Apache Airflow orchestration guide are the natural host for both pre-ingestion gates and circuit breakers, because they already model tasks, dependencies, and failure propagation. A validation task placed upstream of a load task allows the orchestrator to skip or fail the load automatically when the check does not pass, so the circuit-breaker behavior falls out of the dependency graph rather than requiring bespoke error handling.

Dead-Letter Queues for Streaming

In a streaming pipeline, halting the entire stream because a single record fails a check would be too blunt, since one malformed message would block every well-formed one behind it. A dead-letter queue (DLQ) solves this by routing records that fail schema or quality checks to a separate queue for inspection and later replay, while valid records continue on the main stream. This isolates failures at the level of the individual record rather than the whole pipeline, and it preserves the malformed records for diagnosis instead of discarding them. Figure 4 shows a stream splitting into a pass branch and a dead-letter branch, with a replay path back into the pipeline once the underlying problem is fixed.

Contracts and Observability: Prevention Versus Detection

Data contracts are frequently discussed alongside data observability, and the two are sometimes conflated. They address the same goal—reliable data—from opposite directions. A contract is a mechanism of prevention: it asserts expectations at the boundary and refuses data that violates them, so a defect is stopped before it enters the system. Observability is a mechanism of detection: it monitors the data that has already landed—tracking freshness, volume, distribution, and schema drift—and raises an alert when reality departs from the norm. Observability finds problems the contract did not anticipate, including gradual distribution shifts and upstream failures that produce technically valid but unusual data. Figure 8 places the two side by side over a shared pipeline.

Neither mechanism substitutes for the other. A contract cannot anticipate every failure mode, particularly slow statistical drift in otherwise valid data; observability catches those. Observability, on its own, only tells a team that something has already gone wrong, often after consumers have been affected; a contract prevents the class of failures it can express. The two belong together in a mature stack. Quality gates also interact with the storage layer, because a table’s schema evolution is itself a contract concern—an area explored in the comparison of Iceberg, Delta Lake, and Hudi table formats and in the discussion of typing at the file level in the guide to Parquet and Apache Arrow internals. An end-to-end pipeline where these gates would apply in practice is described in the walkthrough from InfluxDB to AWS Iceberg with Telegraf.

The Business Case for Enforcement

The motivation for this discipline is that poor data quality carries a measurable cost. A frequently cited estimate holds that poor data quality costs organizations at least 12.9 million US dollars per year on average; this figure comes from Gartner’s 2020 Magic Quadrant for Data Quality Solutions, based on a survey of 154 reference customers across 16 vendors (Gartner, as of 2020). It should be read as a 2020 estimate rather than a current measurement, and organizations vary widely, but it captures the order of magnitude at stake. The cost is not only financial. Erroneous data erodes the trust that consumers place in a dataset, and once a dashboard has been visibly wrong, downstream teams begin to second-guess every figure it produces, which slows decisions and encourages the growth of parallel, unofficial data copies. Enforcement at the boundary is a way to protect that trust as much as it is a way to avoid direct cost. Adoption of data contracts is growing as teams formalize the producer-consumer relationship, although a precise adoption figure is not available from a reliable primary survey. The direction of travel is clear from the standards themselves: the consolidation of the Data Contract Specification into ODCS v3.1.0 under the Linux Foundation signals a maturing field converging on shared conventions.

Frequently Asked Questions

What is the difference between a data contract and a database schema?

A database schema declares field names, types, and nullability—the structural shape of a table. A data contract is broader: it wraps the schema together with semantics (what each field means, including units and allowed value sets), a service-level agreement (freshness and availability guarantees), and testable quality expectations, and it is versioned as an explicit agreement between the producing team and its consumers. In short, a schema is one component of a contract, and the contract adds meaning, guarantees, and ownership on top of structure.

Do I need both a schema registry and a tool like Great Expectations?

They cover different concerns and are often used together. A schema registry, such as Confluent Schema Registry, enforces the shape and compatible evolution of messages on a stream, rejecting a producer whose schema is incompatible with the registered version. It does not check semantic quality such as null values in nullable fields, out-of-range numbers, or freshness. A tool like Great Expectations or Soda Core enforces those quality expectations against tables or DataFrames. A streaming platform that also cares about value-level quality therefore benefits from both.

Which data quality dimensions are hardest to enforce automatically?

Accuracy and consistency are the hardest. Accuracy asks whether a value correctly describes the real-world entity it represents, which requires comparison against a trusted system of record rather than a self-contained rule; a value can be well-formed, unique, and within the allowed set yet still be wrong. Consistency asks whether values agree across systems and over time, which requires cross-system reconciliation. Completeness, uniqueness, validity, and timeliness are comparatively straightforward to express as mechanical checks.

Should a team standardize on ODCS or the Data Contract Specification in 2026?

A team beginning in 2026 should standardize on the Open Data Contract Standard (ODCS), currently at version 3.1.0 under the Linux Foundation’s Bitol project. The alternative Data Contract Specification is being deprecated and is converging on ODCS, with its tooling support in the Data Contract CLI and Entropy Data planned only until the end of 2026 (datacontract-specification.com, as of 2026). Choosing the surviving standard avoids a forced migration later.

References

• Open Data Contract Standard (ODCS) v3.1.0 — Bitol / Linux Foundation AI & Data (as of 2026)
• Great Expectations (GX Core) changelog (as of 2026-06-26)
• Soda Core release notes (as of 2026-06-29)
• Confluent Schema Registry — Schema Evolution and Compatibility (as of 2026)
• dbt data tests and unit tests documentation (as of 2026-07-01)
• Pandera documentation (as of mid-2026)

Conclusion

Data reliability is achieved not by a single tool but by a discipline that combines an explicit artifact with layered enforcement. The data contract turns the implicit producer-consumer agreement into versioned, machine-readable code, and the Open Data Contract Standard v3.1.0 provides a vendor-neutral way to express it; the consolidation of the older Data Contract Specification into ODCS marks a field settling on shared conventions. Quality itself becomes testable once it is decomposed into completeness, uniqueness, validity, accuracy, consistency, and timeliness, with the understanding that accuracy and consistency resist mechanical checks and require external reference points.

The contract is then enforced wherever data changes form: Pydantic and Pandera at the record and DataFrame boundary, dbt tests and Great Expectations and Soda Core in the warehouse, and a schema registry on the stream. The patterns that put enforcement into force—shift-left validation at the producer, continuous-integration gates on contract changes, pre-ingestion quarantine, circuit breakers, and dead-letter queues—each trade some availability for correctness in the way best suited to their layer. Finally, contracts and observability are complementary rather than interchangeable: contracts prevent the failures they can express at the boundary, while observability detects the anomalies they cannot anticipate. A platform that runs both, over a contract treated as a first-class artifact, is one whose data consumers can trust.

Apache Parquet and Apache Arrow are the two most widely deployed columnar data formats in modern analytics, yet most engineers who depend on them daily treat them as opaque. A query runs against a Parquet file through Spark, DuckDB, Polars, or pandas, the result returns quickly, and the reasons remain hidden. This post opens the black box. It describes what the bytes of a Parquet file actually look like on disk, how encoding and compression stack into a layered pipeline, how metadata and statistics let a reader skip most of the data it never needs, and how Apache Arrow provides the complementary in-memory format that lets independent engines exchange columns without paying a serialization cost. The goal is a mechanistic understanding that translates directly into tuning decisions: row-group sizing, encoding choice, compression codec, page index, and Bloom filters.

The two projects are designed to work together. Parquet is a space-efficient format optimized for durable storage on disk or in object storage. Arrow is a compute-efficient format optimized for processing in memory. A reader decodes Parquet pages into Arrow arrays, operates on them in a CPU-friendly layout, and hands them to another library through raw memory pointers. Understanding both halves, and the bridge between them, explains why a column-oriented analytical stack behaves the way it does.

Why columnar storage exists

A data file must choose an order in which to write the values of a table. A row-oriented format stores all fields of the first record together, then all fields of the second record, and so on. This layout suits transactional workloads that read or write entire records one at a time. An analytical workload behaves differently. It scans a small number of columns across a very large number of rows, computing aggregates, filters, and joins. For such a query, a row-oriented file forces the reader to touch every field of every record even when only two of forty columns are relevant.

A column-oriented, or columnar, format inverts the layout. It stores all values of the first column together, then all values of the second column, and so on. A query that reads two columns reads only those two contiguous regions and ignores the rest of the file. This property, reading only the requested columns, is called projection pushdown, and it is essentially free in a columnar format because the requested data is already grouped.

Columnar layout produces a second benefit that compounds the first. Values within a single column share a data type and often share statistical structure — a timestamp column tends to increase monotonically, a category column repeats a small set of strings, a sensor column clusters around a mean. Grouping like values together lets the format apply type-aware encodings and lets general-purpose compressors find far more redundancy than they could in an interleaved row layout. The figure below contrasts the two layouts for a small table.

This division of labor is deliberate. A durable storage format optimizes for size, because bytes on disk or in object storage cost money and network transfer time. An in-memory format optimizes for processing speed, because a CPU or GPU benefits from predictable, contiguous, aligned buffers. Apache Parquet fills the first role and Apache Arrow fills the second. The remainder of this post examines each in turn, then the bridge that connects them.

Anatomy of a Parquet file on disk

A Parquet file is a self-describing container built from a strict hierarchy. Understanding that hierarchy is the foundation for every tuning decision that follows, because each level maps to a unit of I/O or a unit of metadata.

The file, row group, column chunk, and page

At the top is the file. A file is divided horizontally into row groups. A row group is a partition of the rows — a contiguous band of records — and it is the largest unit that can be processed independently. Within a row group, each column is stored as a column chunk, which holds every value of that one column for that band of rows. Each column chunk is in turn divided into pages. A page is the smallest unit that must be read and decompressed in full to access any record inside it, and each page is encoded and compressed independently. The default row group size is 128 MB (the parquet.block.size configuration is 134217728 bytes) and the default page size is 1 MB (the parquet.page.size configuration is 1048576 bytes), according to the Parquet configuration documentation (parquet.apache.org, as of 2026-07-02).

The file is framed by a four-byte magic value, PAR1, at both the beginning and the end. The trailing region before the closing magic is the footer, which holds the FileMetaData: the schema, the list of row groups, the metadata and statistics for every column chunk, and the byte offsets that locate optional structures such as the Page Index and Bloom filters. A reader opens a Parquet file by seeking to the end, reading the footer, and using it as a map to the rest of the file. The figure below shows this layout.

The Parquet specification defines this structure in two parts that are read together: a prose format document and a Thrift definition file, parquet.thrift, that specifies the exact metadata structures (apache/parquet-format README, as of 2026-07-02). The latest format specification version is 2.12.0 (apache/parquet-format CHANGES.md, as of 2026-07-02). The format continues to evolve — a Variant type for semi-structured data was announced in February 2026 (parquet.apache.org blog, as of 2026-07-02).

Data page versions

Pages come in several types. A DictionaryPage stores the distinct values of a dictionary-encoded column. A DataPage carries the actual values. There are two versions of the data page. The original DataPage (V1) stores repetition levels, definition levels, and encoded values together in one compressed block. DataPageV2 separates the repetition and definition levels from the values, allows those levels to remain uncompressed while the values are compressed, and gives per-page control over whether compression is applied. This separation is what unlocks the delta encoding family described in the next section. An IndexPage type is reserved in the specification. The figure below contrasts the two data page versions.

Repetition and definition levels are the mechanism by which Parquet represents nested and repeated fields in a flat columnar store. This technique originates in the Dremel system described by Melnik and colleagues (VLDB 2010), the foundational paper for Parquet’s nested encoding. A definition level records how many of a column’s optional or nested ancestors are actually present for a given value, which is how nulls at any nesting depth are represented without storing placeholder values. A repetition level records at which nesting level a repeated field begins a new list. Together the two levels let a reader reconstruct arbitrarily nested records from flat columns.

Encoding and compression

Two distinct byte-reduction layers operate inside a Parquet column chunk, and conflating them is a common source of confusion. Encoding is a structural, format-aware transformation applied to the values of one column. Compression is a byte-level, general-purpose transformation applied to the already-encoded bytes of a page. The two stack: values are first encoded, then the encoded output is compressed. A reader reverses the order, decompressing a page and then decoding its values.

The encoding families

PLAIN encoding is the baseline. It writes values in their natural fixed-width or length-prefixed form with no attempt at reduction, and it exists as a fallback when no structural pattern can be exploited.

Dictionary encoding is the workhorse for columns with a limited number of distinct values, such as country codes, status strings, or category labels. The writer builds a dictionary of distinct values, stored in a DictionaryPage, and replaces each value in the data pages with a small integer index into that dictionary. Those indices are then written with the RLE_DICTIONARY encoding, which has enum value 8 in the specification, is valid for all physical types, and is the default for string columns (apache/parquet-format Encodings.md, as of 2026-07-02). If the dictionary grows beyond the configured dictionary page size — the default dictionary page size is 1 MB (parquet.apache.org configurations, as of 2026-07-02) — the writer abandons the dictionary for that column chunk and falls back to PLAIN. One dictionary page is written per column per row group.

The RLE / bit-packing hybrid encoding underlies dictionary indices and also carries the repetition and definition levels. It combines run-length encoding, which represents a repeated value as a value-plus-count pair, with bit-packing, which stores small integers using only as many bits as their range requires rather than a full 32 or 64 bits. The encoder switches between the two modes according to which is smaller for each run.

The delta encoding family targets numeric and byte-array columns and is a DataPageV2 feature, not used for V1 pages (parquet.apache.org encodings, as of 2026-07-02). DELTA_BINARY_PACKED, enum value 5, applies to INT32 and INT64 columns and stores the differences between consecutive values rather than the values themselves (apache/parquet-format Encodings.md, as of 2026-07-02). For a sorted or monotonic column such as a timestamp or an auto-incrementing identifier, those differences are small and highly repetitive, so the encoded output is far smaller than the raw integers. DELTA_BYTE_ARRAY, enum value 7, applies to BYTE_ARRAY and FIXED_LEN_BYTE_ARRAY columns; it encodes the lengths of the byte arrays using DELTA_BINARY_PACKED and then concatenates the bytes, and it is preferred over PLAIN for byte arrays (parquet.apache.org encodings, as of 2026-07-02). A related variant, DELTA_LENGTH_BYTE_ARRAY, separates the length stream from the data stream for strings without shared prefixes.

BYTE_STREAM_SPLIT, enum value 9, applies to INT32, INT64, FLOAT, DOUBLE, and FIXED_LEN_BYTE_ARRAY columns (apache/parquet-format Encodings.md, as of 2026-07-02). It does not reduce size by itself. Instead it scatters the K bytes of each value into K separate streams, placing all the first bytes together, then all the second bytes, and so on. Floating-point values that cluster around a common magnitude share high-order bytes, and grouping those bytes together gives the downstream compressor much more redundancy to exploit (parquet.apache.org encodings, as of 2026-07-02). This is a clear illustration of the encoding-then-compression stack: the encoding rearranges bytes so that the compression layer performs better.

The compression codecs

After encoding, each page is compressed with a block-level, general-purpose codec. The supported codecs are UNCOMPRESSED, SNAPPY, GZIP, LZ4 and LZ4_RAW, ZSTD, and BROTLI. Because each page is compressed independently, a reader can decompress exactly the pages a query touches and no others. The choice among codecs is a trade-off between compression ratio and processing speed, and the appropriate choice is workload-dependent rather than fixed.

Metadata, statistics, and pushdown

The performance of an analytical scan depends less on how fast the reader processes data than on how much data it manages to avoid reading at all. Parquet supports this avoidance through metadata stored in the footer and in optional index structures. Two mechanisms operate together: projection pushdown selects which columns to read, and predicate pushdown selects which rows — more precisely, which row groups and pages — to read.

Projection pushdown was described earlier. Because each column is stored as an independent column chunk, a reader consults the footer to find the byte ranges of the requested columns and reads only those ranges. No special index is required; the columnar layout makes it free.

Predicate pushdown is more involved and operates in three pruning tiers, each addressing a case the previous tier cannot handle. The figure below shows the tiers as a funnel that progressively narrows the data a query must read.

The three pruning tiers

The first tier is row-group statistics. The footer stores, for each column chunk, the minimum value, the maximum value, and the null count. When a query filters on a column, the reader compares the predicate against these statistics. If a row group’s minimum and maximum for the filtered column cannot possibly satisfy the predicate, the reader skips the entire row group — up to 128 MB of data — without reading any of its pages. This tier is coarse but very cheap, since the statistics are already in the footer the reader parsed on open.

The second tier is the Page Index, which brings the same idea down to page granularity. The Page Index was introduced in Parquet 1.11 and is used by Spark 3.2.0 and later (CERN Databases blog and parquet-format PageIndex.md, as of 2026-07-02). It consists of two structures. The ColumnIndex stores the minimum and maximum values for each page in a column chunk, letting the reader identify exactly which pages could contain matching rows. The OffsetIndex maps a row index to the byte offset of the page that contains it, so once the ColumnIndex has selected the relevant pages, the reader seeks directly to them. Together these let a reader skip individual pages within a column chunk that survived the row-group filter.

The third tier is the Bloom filter, which handles a case the range-based tiers cannot. Minimum and maximum statistics prune well when the filtered values fall outside a range, but they are useless for equality predicates on high-cardinality columns whose values are scattered across the whole domain. A filter such as user_id = 90422 will match the min/max range of almost every row group even though only a handful of rows qualify. A Bloom filter is a compact probabilistic structure that answers set membership. Parquet supports exactly one representation, the Split Block Bloom Filter (SBBF), whose block size is 256 bits, arranged as eight contiguous 32-bit words (apache/parquet-format BloomFilter.md, as of 2026-07-02). It answers a membership query with either “definitely not present” or “probably present” — it produces no false negatives, only occasional false positives (apache/parquet-format BloomFilter.md, as of 2026-07-02). When the answer is “definitely not present,” the reader skips the row group. Dictionary pages can serve a similar membership-pruning role, since a value absent from a column chunk’s dictionary cannot appear in that chunk.

Apache Arrow and the Parquet-to-Arrow bridge

Parquet describes data at rest. Once a reader decodes a Parquet page, the values must live somewhere in memory in a form suitable for computation. That form is Apache Arrow. Arrow defines a language-independent, in-memory columnar layout for both flat and nested data, designed for efficient analytics on CPUs and GPUs (arrow.apache.org Columnar.html, as of 2026-07-02). The latest Apache Arrow release is 24.0.0, published on 2026-04-21, comprising 259 resolved issues and 325 commits from 57 contributors over roughly three months (arrow.apache.org blog, as of 2026-07-02).

Where Parquet optimizes for size on disk, Arrow optimizes for access speed in memory. An Arrow array stores a column’s values in a contiguous buffer with a separate validity bitmap indicating which entries are null. This layout is predictable and aligned, so a processor can iterate over it sequentially and apply vectorized instructions that operate on many values per cycle. The Parquet-to-Arrow bridge decodes Parquet pages directly into Arrow arrays: a library such as pyarrow reads a Parquet file and produces Arrow Tables, translating the on-disk encodings into the in-memory layout in one step.

Zero-copy and the C Data Interface

The most consequential property of Arrow is that independent libraries can agree on the exact byte layout of a column and therefore share it without copying. When a query engine, a dataframe library, and a machine-learning framework all represent a column as the same Arrow buffer, one can hand the column to another by passing a pointer rather than serializing, transmitting, and deserializing the data. This is the mechanism the Arrow C Data Interface formalizes. It enables zero-copy sharing between runtimes within a single process through memory pointers, and its only dependency is a stable C application binary interface (ABI) — a fixed set of C structures that every participant understands (arrow.apache.org CDataInterface.html, as of 2026-07-02).

This capability is why an in-process analytical stack can combine tools that were developed independently without incurring a translation tax between them. A query engine can execute a filter, hand the resulting Arrow columns to a dataframe library for a transformation, and pass those to a machine-learning framework, all without any of the three re-serializing the data. The comparison of in-process engines that read Parquet and exchange Arrow buffers is examined in the companion post on DuckDB versus Polars for in-process analytics.

Three ways Arrow data moves

Arrow data is exchanged through several mechanisms, and distinguishing them prevents confusion. The C Data Interface moves data within a single process by passing pointers to buffers already in memory. Arrow IPC, also known as the Feather file format, serializes Arrow data into a byte stream for transmission across a network or for writing to a file — it is a wire and storage format for Arrow itself. Parquet, finally, is the durable at-rest format optimized for long-term storage rather than for immediate computation. Each serves a different boundary: within a process, across a process or network, and at rest. The figure below maps the three mechanisms to the boundaries they cross.

Table formats such as Apache Iceberg, Delta Lake, and Apache Hudi layer a metadata and transaction management system on top of Parquet data files rather than replacing them; the distinction between the file layer and the table layer is examined in the comparison of Iceberg, Delta Lake, and Hudi table formats. A transformation tool that materializes analytical models frequently writes those models as Parquet, a pattern covered in the guide to dbt transformation pipelines.

Practical guidance for tuning

The internals described above translate into a small set of tuning decisions. None of them has a single correct answer; each depends on the storage medium, the reader’s memory budget, and the query patterns.

Row-group sizing

The default row-group size is 128 MB (parquet.apache.org configurations, as of 2026-07-02). A larger row group amortizes footer and metadata overhead across more rows and produces longer contiguous reads, which suits object storage where each request has latency. A smaller row group increases the granularity of row-group statistics pruning and reduces the memory a reader must hold to process one row group. When files are read from object storage, aligning the row-group size with the object store’s block or multipart size reduces the number of range requests. When a reader has a constrained memory budget, a smaller row group prevents it from having to buffer an entire 128 MB band per column.

Sorting to help encoding and pruning

Sorting the data by a frequently filtered column improves two mechanisms at once. First, a sorted numeric column has small, repetitive deltas, which makes DELTA_BINARY_PACKED encoding highly effective. Second, sorting tightens the minimum and maximum ranges within each row group and page, so that row-group statistics and the Page Index prune far more aggressively — an unsorted column tends to span the full value range in every row group, defeating range-based pruning. Sorting therefore benefits both file size and scan speed.

When to enable Bloom filters

A Bloom filter earns its space only on a high-cardinality column that is queried with equality predicates, such as a user identifier, a session token, or an email address. On such a column, minimum and maximum ranges overlap across every row group and cannot prune, so the Split Block Bloom Filter provides the only membership-based pruning available. On a low-cardinality column, a Bloom filter is wasteful, because dictionary encoding and range statistics already prune those columns effectively. Bloom filters should be enabled selectively, per column, based on query patterns.

Choosing a codec and page version

For hot data queried frequently, Snappy’s fast decompression favors query latency. For cold or archival data, Gzip’s higher ratio reduces storage cost at the expense of slower reads. Zstd offers a tunable middle ground and is increasingly chosen as a default because its ratio approaches Gzip’s at speeds closer to Snappy’s; the exact ratio remains workload-dependent and should be measured on the real dataset. DataPageV2 is required to use the delta encoding family, so writers that store sorted integer or timestamp columns benefit from enabling V2 pages. These file-level choices are typically set once in the writer configuration of the job that produces the data, whether that job runs under an orchestrator such as the one described in the guide to Apache Airflow pipeline orchestration or as part of an ingestion pipeline like the one in the walkthrough of an InfluxDB-to-Iceberg data pipeline.

Frequently Asked Questions

What is the difference between Apache Parquet and Apache Arrow?

Parquet is a format for storing columnar data on disk or in object storage, optimized for space efficiency through encoding and compression. Arrow is a format for holding columnar data in memory, optimized for fast processing on CPUs and GPUs. A reader decodes Parquet files into Arrow arrays to compute on them. The two are designed together: Parquet handles data at rest and Arrow handles data in memory.

What is the default row-group and page size in Parquet?

The default row-group size is 128 MB, set by the parquet.block.size configuration at 134217728 bytes, and the default page size is 1 MB, set by parquet.page.size at 1048576 bytes (parquet.apache.org configurations, as of 2026-07-02). A row group is a horizontal band of rows processed independently, and a page is the smallest unit that must be read and decompressed in full to access any record inside it.

How does predicate pushdown skip data in a Parquet file?

Predicate pushdown prunes data in three tiers. Row-group statistics in the footer store per-column min, max, and null count, letting a reader skip an entire row group. The Page Index refines this to individual pages using per-page min and max values with a row-index-to-offset map. The Split Block Bloom Filter handles equality on high-cardinality columns where ranges cannot prune, answering “definitely not present” or “probably present” (apache/parquet-format BloomFilter.md, as of 2026-07-02).

Why do Parquet files need both encoding and compression?

Encoding and compression are two separate layers that stack. Encoding is a structural, format-aware transformation of one column’s values, such as dictionary encoding or delta encoding, that exploits the column’s type and distribution. Compression is a general-purpose byte-level codec, such as Snappy or Zstd, applied to the already-encoded page. Encoding often rearranges bytes so the compressor finds more redundancy; BYTE_STREAM_SPLIT is a clear example, adding no size reduction itself but improving downstream compression (parquet.apache.org encodings, as of 2026-07-02).

What does the Arrow C Data Interface do?

The C Data Interface lets separate libraries in the same process share Arrow column buffers without copying, by passing pointers through a small set of standard C structures. Its only requirement is a stable C application binary interface (arrow.apache.org CDataInterface.html, as of 2026-07-02). This lets a query engine, a dataframe library, and a machine-learning framework exchange columns without serializing and deserializing the data at each handoff.

References

• Apache Parquet format specification and parquet.thrift — apache/parquet-format, spec version 2.12.0 (as of 2026-07-02)
• Parquet encodings reference — parquet.apache.org documentation (as of 2026-07-02)
• Parquet Split Block Bloom Filter specification — apache/parquet-format BloomFilter.md (as of 2026-07-02)
• Apache Arrow columnar format specification — arrow.apache.org (v24.0.0, as of 2026-07-02)
• Apache Arrow C Data Interface — arrow.apache.org (v24.0.0, as of 2026-07-02)
• Dremel: Interactive Analysis of Web-Scale Datasets — Melnik et al., VLDB 2010, the foundational source for Parquet’s repetition/definition-level nested encoding

Conclusion

A Parquet file is not an opaque blob but a carefully layered structure whose every level serves a purpose. The file-to-row-group-to-column-chunk-to-page hierarchy makes projection pushdown free and gives predicate pushdown its units of pruning. The encoding layer exploits the type and distribution of each column, and the separate compression layer removes the byte-level redundancy that remains. The footer’s statistics, the Page Index, and the Split Block Bloom Filter combine into a three-tier funnel that lets a reader avoid the majority of the data a query does not need. Apache Arrow completes the picture by defining the in-memory layout into which those pages decode, and its C Data Interface lets independent libraries share columns without paying a serialization cost.

These internals are not academic. Row-group sizing, sorting by a filtered column, selective Bloom filters, DataPageV2 for delta encodings, and codec choice are all decisions that follow directly from understanding the mechanism. An engineer who knows what the bytes look like can reason about why a scan is slow and what to change, rather than treating the format as a black box. The formats continue to evolve — Parquet at specification version 2.12.0 with a new Variant type announced in February 2026, and Arrow at release 24.0.0 from April 2026 (as of 2026-07-02) — but the columnar principles described here remain the stable foundation beneath every analytical query that touches them.

DuckDB and Polars have become two of the most widely adopted tools for analytical work that runs inside a single application process, rather than against a separate database server or a distributed cluster. Both read columnar files such as Parquet directly, both execute queries over an in-memory layout derived from the Apache Arrow format, and both are fast enough that many data teams now reach for one of them before considering a cluster engine such as Apache Spark. They are, however, built around different ideas: DuckDB is a relational database that speaks SQL, while Polars is a DataFrame engine driven by an expression API and written in Rust. This post compares the two as engineering tools — their execution models, their memory behavior under load, the benchmark evidence that exists (and its limits), and the often-overlooked fact that they can hand data to each other with no copy. The goal is not to crown a single winner, because the two trade the performance lead depending on scale and workload, but to give a data engineer a clear basis for choosing one, the other, or both.

Two engines, one process: what “in-process analytics” means

The term in-process describes software that runs within the address space of the calling application rather than as a separate service that the application contacts over a network or a socket. A traditional analytical database such as a data warehouse runs as a server: an application opens a connection, sends SQL over the wire, and receives rows back. An in-process engine instead loads as a library inside the program — a Python interpreter, for example — and operates on data in the same memory the program already holds. There is no server to start, no port to manage, and no connection pool to tune.

DuckDB describes itself as “an in-process analytical (OLAP) database.” OLAP, or online analytical processing, refers to read-heavy workloads that scan and aggregate large numbers of rows — the opposite of online transaction processing (OLTP), which favors many small, indexed reads and writes. Polars describes itself as “an analytical query engine written for DataFrames.” A DataFrame is a two-dimensional table abstraction, familiar from libraries such as pandas, in which columns carry typed values and operations are expressed as transformations of whole columns. The two descriptions point at the same workload — analytical queries over tabular data — approached from a relational direction (DuckDB) and a DataFrame direction (Polars). (Sources: duckdb.org/why_duckdb; pypi.org/project/polars, as of 2026-06-22.)

The distinction from a server architecture matters for how a system is built and operated. The figure below contrasts the two deployment shapes.

Neither engine is a distributed system, and this is the single most useful framing for someone deciding when to use them. Apache Spark, the dominant cluster engine for large-scale processing, distributes work across many machines coordinated by a driver and scheduler. That model is appropriate when a dataset genuinely does not fit on one machine or when a workload must be elastic across a fleet. DuckDB and Polars instead exploit a single machine well — every core, the full memory hierarchy, and fast local or cloud storage — and increasingly that is sufficient. Cloud instances now offer hundreds of gigabytes of RAM, and many analytical datasets are smaller than teams assume. When a job fits comfortably on one node, an in-process engine removes the operational weight of a cluster while often finishing faster, because no time is lost to network shuffles or task scheduling.

Because both engines sit at the compute layer, they read from the storage and table-format layer below them rather than replacing it. A team standardizing on open table formats may want to review how those formats differ in the comparison of Apache Iceberg, Delta Lake, and Hudi table formats, since DuckDB and Polars are the engines that would read tables stored in those formats.

Architecture compared: SQL versus DataFrame, eager versus lazy

The two engines share a foundation and diverge above it. Both are columnar, meaning they store and process data column by column rather than row by row. Column-oriented layout suits analytical queries because such queries typically touch a few columns across many rows; reading only the needed columns reduces input/output and lets the processor work on long, uniform runs of values. Both are also vectorized: instead of evaluating one value at a time, they process a batch of values — a vector — in each step of execution, which keeps the processor’s pipeline full and amortizes the overhead of interpreting each operation. DuckDB states that in its engine “queries are still interpreted, but a large batch of values (a ‘vector’) are processed in one operation.” Polars lists “SIMD” operations, referring to Single Instruction, Multiple Data, a processor capability that applies one instruction to several data elements at once. (Sources: duckdb.org/why_duckdb; pypi.org/project/polars, as of 2026-06-22.)

SQL engine versus DataFrame engine

The most visible difference is the interface. DuckDB executes SQL, the declarative query language used by relational databases for five decades. A user writes what result is wanted and the engine decides how to produce it. DuckDB also provides “ACID guarantees through our custom, bulk-optimized Multi-Version Concurrency Control (MVCC)” — ACID being the set of transaction properties (atomicity, consistency, isolation, durability) that keep concurrent changes correct, and MVCC being a technique that lets readers and writers proceed without blocking each other by keeping multiple versions of a row. (Source: duckdb.org/why_duckdb, as of 2026-06-22.)

Polars exposes “a DataFrame expression API plus a SQL interface.” Its primary interface is the expression API, in which transformations are composed as method calls and column expressions in the host language. Polars is “written in Rust,” uses the “Apache Arrow Columnar Format,” and offers frontends in “Python, Rust, NodeJS, R, [and] SQL.” A DataFrame engine appeals to programmers who prefer composing transformations as code, with the full control flow and tooling of a general-purpose language, over embedding SQL strings. (Source: pypi.org/project/polars, as of 2026-06-22.)

Parallelism is realized differently. DuckDB uses “morsel-driven parallelism,” a scheduling approach in which input data is divided into small, fixed-size chunks called morsels that worker threads pull and process, so that load balances across cores without a rigid up-front partitioning. Polars lists “multi-threaded” execution combined with SIMD. Both aim at the same outcome — saturating all available cores on one machine — through different scheduling strategies. (Sources: duckdb.org/why_duckdb; pypi.org/project/polars, as of 2026-06-22.)

Eager versus lazy evaluation

Polars makes an explicit distinction that shapes how performant code is written: eager versus lazy evaluation. In eager mode, each operation runs immediately and returns a materialized result, which is convenient for interactive exploration. In lazy mode, operations build a query plan that is not executed until a terminal call — collect() — is reached. Deferring execution lets the optimizer rewrite the whole plan before any work happens: it can push filters down to the data source so that fewer rows are read, prune unused columns, and reorder operations. Polars documents both modes as “Lazy | Eager execution” with “Query optimization.” (Source: pypi.org/project/polars, as of 2026-06-22.)

DuckDB does not expose an eager/lazy switch because SQL is declarative by nature: a complete statement is handed to the optimizer, which always sees the whole query before execution. The lazy mode of Polars is, in effect, a way to recover that whole-query view inside an imperative DataFrame API. For a data engineer, the practical guidance is that Polars code intended for production should use the lazy API, since eager chains forgo the optimizations that make the engine fast on large inputs.

Streaming and larger-than-memory execution

A defining concern for single-node engines is what happens when a dataset is larger than available RAM. Polars supports processing data “in a streaming fashion,” enabled with collect(engine='streaming'), which executes the query in chunks so that the full dataset need not be resident at once. DuckDB, as a database, has long been designed to spill intermediate state to disk when memory is exhausted, allowing queries to complete on data larger than memory. Both engines therefore offer a path beyond the RAM ceiling, though, as the benchmark section shows, their default memory footprints under stress can differ substantially. (Sources: pypi.org/project/polars; duckdb.org/why_duckdb, as of 2026-06-22.)

Both engines build on the Apache Arrow columnar memory format, an open standard for representing tabular data in memory so that different tools can share buffers without reformatting. Arrow is the reason the two engines interoperate so cheaply, a point developed in the interoperability section. The same Parquet files — Parquet being the dominant open columnar file format on disk — can be read directly by either engine, which means a team is not locked into a proprietary storage format by either choice.

Performance and memory: what the benchmarks actually show

Performance comparisons between DuckDB and Polars are widely circulated and widely misread. Two public benchmarks are credible enough to discuss, and both must be read with their conditions in mind. The first is a throughput benchmark published by the Polars project; the second is a memory-focused stress test from an independent consultancy. No vendor-neutral, audited TPC-H comparison of the two engines existed as of writing, so these are the most credible public figures available, and neither should be treated as the final word.

The PDS-H throughput benchmark

The Polars project publishes a benchmark based on PDS-H, a workload derived from TPC-H. TPC-H is a long-standing industry decision-support benchmark consisting of analytical queries over a synthetic dataset whose size is set by a “scale factor” (SF); SF-10 is roughly ten gigabytes of raw data and SF-100 roughly one hundred. The PDS-H total execution times reported by the Polars project (May 2025) are shown below.

Two patterns stand out. First, both DuckDB and Polars are an order of magnitude faster than pandas, PySpark, and Dask across these scales, which supports the broader claim that an in-process columnar engine outperforms both a single-threaded DataFrame library and a cluster engine on a single node at these sizes. Second, the lead changes with scale: Polars streaming was fastest at SF-10 (3.89 s versus DuckDB’s 5.87 s), while DuckDB was fastest at SF-100 (19.65 s versus Polars streaming’s 23.94 s). The collapse of Polars in-memory from 9.68 s at SF-10 to 152.27 s at SF-100 also shows why the streaming engine matters as data grows. (Source: pola.rs/posts/benchmarks, as of 2026-06-22.)

The Parquet memory stress test

Throughput is only one axis; peak memory determines whether a job fits on a given machine at all. An independent benchmark published on the codecentric blog (published 2026-01-20, updated 2026-02-02) scaled a single Parquet file from roughly 2 GB to 140 GB and measured both execution time and peak memory. On execution time the two engines were very similar across the range, with DuckDB roughly one second faster at the largest scale. The more revealing result was peak memory at 140 GB, summarized below.

Two findings deserve emphasis. First, default behavior diverged by more than an order of magnitude: DuckDB held about 1.3 GB while default Polars used about 17 GB on the same 140 GB file. Second, the gap was not intrinsic — forcing asynchronous reads brought Polars down to about 750 MB, below DuckDB’s footprint. Separately, partitioning the 140 GB dataset into 72 smaller files cut DuckDB’s memory by roughly eight times and default Polars’ by roughly four times, showing that file layout strongly influences memory regardless of engine. (Source: codecentric.de blog, as of 2026-06-22.)

The combined reading of the two benchmarks is consistent with the framing that neither engine is universally faster. On throughput the lead trades by scale; on memory DuckDB is more conservative out of the box, but Polars can match or beat it once tuned, and file layout dominates both. For a team selecting between data stores and engines more broadly, the same “measure under your own conditions” discipline applies; the comparison of databases for preprocessed time-series data walks through a similar evaluation for a related class of workload.

Interoperability: zero-copy handoff over Apache Arrow

A point often missing from “DuckDB versus Polars” discussions is that the two are not mutually exclusive within one program. Because both represent data in the Apache Arrow columnar format, they can share the same in-memory buffers without copying. The official DuckDB documentation states that “DuckDB can read Polars DataFrames and convert query results to Polars DataFrames. It does this internally using the efficient Apache Arrow integration.” In practice, a Polars DataFrame held in a variable can be referenced directly by name inside a SQL query — duckdb.sql("SELECT * FROM df") — and the result can be converted back to a Polars DataFrame with .pl(), or to a Polars LazyFrame with .pl(lazy=True). The pyarrow package is required for this path. (Source: duckdb.org/docs/current/guides/python/polars.html, as of 2026-06-22.)

Zero-copy means the handoff transfers ownership of, or a reference to, the existing memory buffer rather than serializing and reallocating the data. The figure below shows a single Arrow buffer being read by both a Polars DataFrame and a DuckDB query.

This interoperability changes the decision calculus. A pipeline can perform DataFrame-style feature preparation in Polars, hand the frame to DuckDB for a complex multi-table SQL join and aggregation, and receive the result back as a Polars frame for the next step — all within one process and without paying a serialization cost at each boundary. The two engines become complementary stages rather than competing choices.

Both engines also fit cleanly into the wider data ecosystem. They read Parquet from local disk and from object storage such as Amazon S3, which is the common substrate for analytical data lakes. DuckDB has an adapter, dbt-duckdb, that lets it serve as the execution engine for transformation models; teams already using that framework can read about the model-based workflow in the guide to building transformation pipelines with dbt. An in-process engine is also a natural transform step inside a scheduled workflow, where an orchestrator triggers the job; the patterns for that are covered in the guide to orchestrating data pipelines with Apache Airflow. When the analytical layer is fed by streaming ingestion, the upstream side is described in the guide to change data capture with Debezium and Kafka.

Choosing: when DuckDB, when Polars, when both

Because the engines trade the performance lead by scale and workload, the most reliable basis for choosing is fit — to the team’s skills, the surrounding system, and the shape of the work — rather than a single benchmark number. The decision flow below summarizes the practical signals.

The table below condenses the same guidance into a side-by-side reference.

A reasonable default is to choose by interface and team skills first: a team comfortable in SQL, or one that wants transactional guarantees and conservative memory out of the box, will find DuckDB the lower-friction option, while a team writing DataFrame-style transformation code, preparing features for machine learning, or working in Rust will find Polars more natural. When a single pipeline genuinely contains both shapes of work, the zero-copy Arrow bridge makes “use both” a sound engineering choice rather than a compromise. The one case where neither is the right answer is data that cannot be made to fit on a single large machine even with streaming or spilling; that workload still belongs on a distributed engine.

Frequently Asked Questions

Is DuckDB or Polars faster?

Neither is faster in all cases. In the Polars-published PDS-H benchmark (May 2025), Polars streaming was fastest at scale factor 10 (3.89 s versus DuckDB’s 5.87 s) while DuckDB was fastest at scale factor 100 (19.65 s versus 23.94 s). In the independent codecentric Parquet stress test, execution times were very similar across scales, with DuckDB about one second faster at 140 GB. The lead trades by scale and workload, and both benchmarks carry caveats: the PDS-H figures are vendor-published and not comparable to certified TPC-H results, and the stress test is from a consultancy blog. (Sources: pola.rs/posts/benchmarks; codecentric.de blog, as of 2026-06-22.)

Can DuckDB and Polars be used together?

Yes. Both represent data in the Apache Arrow columnar format, so they exchange data zero-copy. A Polars DataFrame in scope can be queried directly by variable name in SQL, for example duckdb.sql("SELECT * FROM df"), and a DuckDB result converts back to a Polars DataFrame with .pl() or to a LazyFrame with .pl(lazy=True). The pyarrow package is required. (Source: duckdb.org/docs/current/guides/python/polars.html, as of 2026-06-22.)

What does “in-process” mean, and how is it different from Spark?

In-process means the engine runs as a library inside the application’s own process rather than as a separate server. DuckDB and Polars both run on a single machine and read files directly, so they remove the operational overhead of a server or a cluster. Apache Spark is a distributed engine that coordinates work across many machines; it is appropriate when data does not fit on one node or when elasticity across a fleet is required. For workloads that fit on one machine, an in-process engine is often simpler and faster.

Why does Polars distinguish eager and lazy evaluation?

In eager mode each operation runs immediately, which suits interactive exploration. In lazy mode operations build a query plan that is executed only at a collect() call, which lets the optimizer rewrite the whole plan first — pushing filters down, pruning unused columns, and reordering steps. Production Polars code generally uses the lazy API to gain these optimizations. DuckDB has no equivalent switch because SQL is declarative and the optimizer always sees the complete query. (Source: pypi.org/project/polars, as of 2026-06-22.)

Which uses less memory?

In the codecentric stress test on a 140 GB single Parquet file, DuckDB used about 1.3 GB by default while default Polars used about 17 GB, but forcing asynchronous reads brought Polars down to about 750 MB. Partitioning the dataset into 72 files cut DuckDB’s memory roughly eightfold and default Polars’ roughly fourfold. DuckDB is more conservative out of the box, but Polars is tunable, and file layout influences memory strongly for both. (Source: codecentric.de blog, as of 2026-06-22.)

References

• DuckDB — Why DuckDB (in-process OLAP, columnar-vectorized execution, MVCC, morsel-driven parallelism), as of 2026-06-22.
• Polars — PyPI project page (analytical query engine for DataFrames, Apache Arrow format, lazy/eager execution, SIMD, streaming), as of 2026-06-22.
• Polars — PDS-H benchmark (TPC-H-derived, May 2025; results not comparable to certified TPC-H), as of 2026-06-22.
• DuckDB documentation — Integration with Polars (zero-copy Apache Arrow handoff, .pl() conversion), as of 2026-06-22.
• DuckDB 1.5.4 release announcement (current release and 1.4.5 LTS line), as of 2026-06-22.

Conclusion

DuckDB and Polars represent two routes to the same destination: fast analytical computation inside a single process, over open columnar data, on one machine. DuckDB approaches the problem as a relational database — SQL, ACID transactions through MVCC, conservative default memory, and morsel-driven parallelism — while Polars approaches it as a Rust DataFrame engine with an expressive expression API, explicit lazy evaluation, and SIMD-accelerated multi-threading. The public benchmarks confirm that neither holds a universal performance lead; the Polars-published PDS-H figures show the lead changing between scale factor 10 and 100, and the independent codecentric stress test shows DuckDB conservative on memory by default while Polars matches it once tuned, with file layout dominating both. Because both build on Apache Arrow, the most consequential and least discussed fact is that they interoperate zero-copy, which turns an apparent rivalry into a pairing. A sound selection therefore rests on interface preference, memory behavior, and ecosystem fit rather than on a single throughput number — and where a pipeline contains both relational and DataFrame work, the right answer is often to use both, bridged over Arrow.

Open table formats turned commodity object storage into a transactional database layer, and the choice among the three principal implementations — Apache Iceberg, Delta Lake, and Apache Hudi — is one of the foundational decisions in a modern lakehouse. The decision is consequential because the table format governs how data is written, updated, queried, and shared across every engine that touches the storage layer. It is also a decision that has changed character over the past two years. Where engineers once treated the choice as a long-term lock-in, the three formats have begun to converge toward interoperability, so that the question is increasingly about defaults and operational fit rather than permanent commitment.

This article examines what a table format is and why it became necessary, how each of the three formats is designed internally, how they compare across the dimensions that matter in practice, and why the so-called format war is giving way to a model in which one physical dataset can be read as more than one format. It closes with concrete guidance on how to choose in 2026.

Why Table Formats Exist

A data lake in its simplest form is a collection of files — most commonly Apache Parquet, a columnar file format — stored in an object store such as Amazon S3, Google Cloud Storage, or Azure Blob Storage. A query engine reads those files and treats them collectively as a table. This arrangement is inexpensive and scalable, but the abstraction is weak. The storage layer knows only about files and directories; it has no concept of a table, a transaction, or a consistent snapshot. The problems that follow from this gap are what gave rise to table formats.

A table format is a specification and metadata layer that sits between a physical collection of data files and the query engines that read them, presenting that collection as a single logical table with database-like guarantees. The most important of those guarantees is ACID — atomicity, consistency, isolation, and durability — the set of properties that ensure a group of changes either fully applies or does not apply at all, that concurrent readers and writers do not observe partial state, and that committed data survives failures. Without a table format, the lake cannot offer these properties.

Consider what happens when a job appends a new day of data to a plain Parquet directory and fails halfway. Some files are written and some are not, and a reader that lists the directory during the failure observes an inconsistent table. The same fragility affects updates and deletes: changing a single record requires rewriting whole files, and there is no transactional boundary to make the change appear atomically. This is the “directory of files” problem — the table is defined by whatever files happen to be present when the engine lists the path, with no authoritative record of which files belong to a committed version.

Three further capabilities are absent from the plain-Parquet model and are central to why table formats were created. Schema evolution is the ability to add, rename, drop, or reorder columns without rewriting historical data or breaking existing readers. Time travel is the ability to query the table as it existed at a previous point in time, which supports reproducibility, auditing, and rollback. And efficient query planning depends on metadata that records which files contain which ranges of values, so the engine can skip files that cannot contain matching rows.

Table formats deliver these capabilities through a metadata layer — a set of files that record the table’s schema, its partitioning, and, critically, the exact list of data files that constitute each committed version of the table. A central concept in two of the three formats is the manifest: a metadata file that enumerates a group of data files together with statistics about their contents, such as the minimum and maximum value of each column within each file. With this information the engine performs file pruning, reading only the files that could contain rows matching a query predicate. The three formats examined below implement these ideas differently, and those differences drive their respective strengths.

Apache Iceberg: Hierarchical Metadata and Engine Neutrality

Apache Iceberg was created to provide a table format that no single engine owns and that scales to very large tables without the planning bottlenecks of earlier approaches. Its defining characteristic is a hierarchical metadata architecture and a specification-first design philosophy. The format is defined by a written specification, and any engine that implements the specification can read and write the tables, which is the basis for Iceberg’s wide adoption.

The metadata hierarchy

An Iceberg table is described by a tree of metadata files. At the root is a metadata.json file (often called the table metadata file) that records the current schema, partition specification, and a pointer to the current snapshot. Each snapshot references a manifest list, a file that enumerates the manifest files belonging to that snapshot. Each manifest file, in turn, tracks a set of individual data files and stores column-level statistics for them, such as per-column lower and upper bounds and null counts (Apache Iceberg documentation; Dremio, as of 2026).

This hierarchy has two practical consequences. First, query planning is efficient even for tables with millions of files, because the engine reads the manifest list and the relevant manifests rather than listing the object store. Second, because the statistics are recorded in the manifests, the engine can prune files during planning — eliminating files whose recorded value ranges cannot satisfy the query predicate — without opening the data files. This is the same principle that motivates careful storage selection for analytical workloads, a topic explored in the discussion of choosing databases for preprocessed time-series data.

Partition evolution and hidden partitioning

Two features distinguish Iceberg’s treatment of partitioning. Hidden partitioning means the table records how a column is transformed into partition values — for example, by truncating a timestamp to its day — so that queries filtering on the raw column automatically benefit from partition pruning without the writer or reader having to reference a separate partition column. Partition evolution means the partitioning scheme of a table can be changed over time without rewriting the historical data; new data is written under the new scheme while old data remains valid under the old one. This is significant operationally, because it removes one of the most expensive and disruptive migrations in traditional partitioned tables.

Engine reach

Iceberg’s specification-first, engine-agnostic design has produced the broadest multi-engine support of the three formats. As of 2026 it is read and written by Apache Spark, Apache Flink, Trino, and DuckDB, and is supported as a table format by managed warehouses including Snowflake and Google BigQuery (RisingWave; Dremio; Onehouse comparison guides, as of 2026). This breadth is the principal reason Iceberg has become the common denominator across the industry, a point developed in the convergence section. A practical example of landing data into Iceberg from an operational source is described in the guide to building an InfluxDB-to-Iceberg data pipeline with Telegraf, and stream processors such as Flink commonly write into Iceberg as part of complex event processing pipelines.

Delta Lake: A Transaction Log Born at Databricks

Delta Lake originated at Databricks and was designed first and foremost to bring transactional reliability to data stored for use with Apache Spark. Its integration with Spark is the deepest of the three formats, reflecting that shared origin (Databricks; Dremio, as of 2026). Where Iceberg organizes metadata as a tree of manifest files, Delta Lake organizes it as an ordered transaction log.

The transaction log

A Delta table keeps a directory named _delta_log alongside its data files. Each successful commit writes a new JSON file to this directory, numbered sequentially, recording the actions that the commit performed — which data files were added and which were removed, along with metadata changes. The current state of the table is obtained by replaying these JSON commits in order. To prevent the replay from growing unbounded, Delta Lake periodically writes a checkpoint in Parquet format that captures the full table state up to a given commit. A reader then loads the most recent checkpoint and applies only the JSON commits that follow it, which keeps state reconstruction efficient.

The transaction log model gives Delta Lake straightforward time travel — a reader can request the table state as of any commit version — and reliable concurrency control through optimistic commits against the log. Because the log is the single ordered authority on table state, the semantics map cleanly onto Spark’s execution model, which is part of why Delta Lake remains the most natural choice within Spark-centric and Databricks-centric environments. Transformation workflows that run on top of such tables, for instance with dbt-based transformation pipelines, treat the table as the consistent input and output of each model run.

UniForm and interoperability

For most of Delta Lake’s history its log-based metadata was readable only by Delta-aware engines, which limited reach relative to Iceberg. Delta UniForm addresses this directly. UniForm provides interoperability across Delta Lake, Iceberg, and Hudi by generating the metadata that other formats expect alongside the Delta log, and it supports the Iceberg REST catalog interface so that Iceberg clients can discover and read the tables (Databricks, as of 2026). In effect, a table written as Delta can be presented to an Iceberg reader without copying the underlying Parquet data. This is one of the developments that has softened the practical cost of choosing Delta in a mixed-engine environment, and it is examined further in the convergence section.

Apache Hudi: Built for Streaming Upserts

Apache Hudi was designed around a workload that the other two formats addressed only later: continuous ingestion of streaming data with frequent record-level updates. The name itself abbreviates “Hadoop Upserts Deletes and Incrementals,” which signals its orientation. As of 2026, Hudi offers the most mature tooling for pure streaming ingestion involving high-frequency upserts (Onehouse; lakeFS, as of 2026). An upsert is an operation that inserts a record if its key does not yet exist and updates the existing record if it does — the natural operation when a change-data-capture stream delivers a steady flow of inserts, updates, and deletes from an operational database. Feeds of this kind commonly arrive through change data capture with Debezium and Kafka or through a Kafka consumer that lands events into the lake.

Copy-on-Write and Merge-on-Read

Hudi offers two table types, and the choice between them is the central tuning decision when adopting the format. In a Copy-on-Write (CoW) table, an update rewrites the base file that contains the affected record, so every version of the data is materialized at write time. Reads are therefore fast and simple, because the engine reads finished base files, but writes are more expensive because they rewrite whole files. In a Merge-on-Read (MoR) table, an update is written to a delta log file associated with the base file rather than rewriting the base file immediately; readers merge the base file with its delta logs at query time. Writes are cheaper and lower-latency, which suits high-frequency upserts, at the cost of more work at read time until a background compaction merges the deltas into new base files.

Record-level indexing and table management

To apply an upsert, Hudi must locate the existing record for a given key quickly. It maintains record-level indexing for this purpose — a mapping from record keys to the files that contain them — so an incoming update can be routed to the correct file without scanning the whole table. This indexing is a meaningful part of why Hudi handles high-frequency update workloads efficiently. Hudi also includes built-in table management services: compaction merges MoR delta logs into base files, clustering reorganizes data to improve query locality, and cleaning removes obsolete file versions according to a retention policy. These services can run as part of the ingestion pipeline or as separate jobs, and orchestrating them alongside ingestion is a common use of a workflow scheduler such as Apache Airflow.

Hudi’s primary ingestion utility is its streaming ingestion tool, historically known as DeltaStreamer (now Hudi Streamer), which reads from sources such as Kafka and applies inserts, updates, and deletes to a Hudi table continuously. This tooling, together with record-level indexing and the MoR table type, is the basis for Hudi’s reputation in upsert-heavy streaming ingestion.

Native Iceberg output

Hudi has also moved toward interoperability. Tables created from Hudi version 0.14.0 onwards can be synced to Iceberg and/or Delta Lake through Apache XTable, and Hudi’s native Iceberg support lets a team use Hudi’s managed services — compaction, indexing, and Hudi Streamer ingestion — while outputting Iceberg-compatible tables for downstream consumers (Apache Hudi documentation, as of 2026). This means a pipeline can keep Hudi’s strengths on the write side while presenting Iceberg on the read side, a pattern that the convergence section places in context.

A Head-to-Head Comparison

The three formats overlap substantially in their core guarantees — all provide ACID transactions, schema evolution, and time travel — so the meaningful differences lie in their metadata models, their maturity for specific operations, and their ecosystem reach. The matrix below summarizes the breadth of engine support that distinguishes the formats, followed by a feature comparison and a workload mapping.

The grid is directional rather than a precise capability audit; engine support changes with each release, and warehouse vendors continue to add native read paths. The pattern it conveys is the one reported consistently across the comparison literature: Iceberg has the widest native reach, Delta Lake is strongest within Spark and reaches other engines primarily through UniForm, and Hudi is strong in Spark and Flink and reaches warehouses through translation. The detailed feature comparison follows.

The feature comparison clarifies that the formats are not interchangeable on every axis even though their core guarantees overlap. Partition evolution and hidden partitioning are genuine Iceberg differentiators; built-in table management is a genuine Hudi differentiator; and the deepest Spark integration remains a Delta Lake characteristic. The second table maps common workloads to the format that most naturally fits each, before interoperability is taken into account.

The Convergence Story

The most consequential change in this area is not a new capability in any single format but the erosion of the boundaries between them. Several developments, taken together, have moved the ecosystem from competition toward interoperability, and they explain why the choice of format is now less of a permanent commitment than it was.

The Databricks–Tabular acquisition

In a development that reframed the competitive landscape, Databricks acquired Tabular — the startup founded by the original creators of Apache Iceberg — for more than one billion dollars, a deal revealed on June 4, 2024 (Databricks blog; TechTarget, as of 2024-06-04). Because Databricks is the company behind Delta Lake, the acquisition brought the architects of the rival format inside the same organization and signaled an intent to support both formats rather than insist on one. It is widely read as the moment the “format war” framing began to lose force in favour of interoperability.

UniForm, Hudi output, and Apache XTable

Three technical mechanisms now allow a single physical dataset to be read as more than one format. Delta UniForm, described earlier, exposes Delta tables as Iceberg and supports the Iceberg REST catalog interface (Databricks, as of 2026). Hudi’s native Iceberg support lets a team manage a table with Hudi and serve it as Iceberg (Apache Hudi documentation, as of 2026). And Apache XTable — an incubating project backed by Microsoft, Google, and Onehouse — provides omni-directional metadata translation between all three formats: any format to any format, with no copying of the underlying data (Dremio; Onehouse, as of 2026). XTable works by generating the metadata that each target format expects, pointing it at the same Parquet files, so the cost of translation is metadata generation rather than data duplication.

The cumulative effect of these developments is that Apache Iceberg has become the de-facto common denominator. It is the format that the other two can be exposed as, the one with the broadest native reach, and the one adopted by every major cloud provider and query engine (RisingWave; Dremio; Onehouse, as of 2026). The reasons most often cited are the same throughout the literature: vendor-neutral governance, partition evolution, and the widest multi-engine support.

How to Choose in 2026

Given convergence, the practical decision reduces to selecting the format that best fits the write path and the operational center of gravity, while relying on interoperability to satisfy diverse readers. The decision tree below summarizes the guidance, and the discussion that follows expands on each branch.

Iceberg as the safe default

For a new deployment that aims to remain vendor-neutral and to be queried by several engines, Iceberg is the safe default. It carries the lowest risk of lock-in, offers partition evolution and hidden partitioning, and is natively supported across the widest range of engines and warehouses. Choosing Iceberg also aligns with the direction of the ecosystem, since the other formats can be exposed as Iceberg but the reverse arrangement is less central to current tooling. For teams running engines on Kubernetes, where portability is already a goal, the neutrality argument extends naturally to the storage layer; the operational considerations of running such engines are discussed in the guide to database connections from Kubernetes pods.

Delta Lake for Databricks-centric stacks

When the platform is built around Databricks and Spark, Delta Lake remains the natural choice. Its integration is the deepest available, its tooling and governance through Unity Catalog are mature, and UniForm now mitigates the historical drawback of limited external reach by exposing the same tables as Iceberg. A team already invested in Databricks gains little by writing Hudi or Iceberg directly and may give up integration depth by doing so.

Hudi for streaming upserts

For ingestion dominated by high-frequency upserts and change-data-capture streams, Hudi remains the strongest fit. Its record-level indexing, Merge-on-Read table type, and built-in compaction and cleaning were designed for exactly this pattern, and its Hudi Streamer utility provides a tested ingestion path. The native Iceberg output then allows the same data to be served to analytical consumers as Iceberg, combining Hudi’s write strengths with Iceberg’s read reach.

When to rely on interoperability instead

Some organizations do not need to pick a single winner. When one dataset must serve consumers that expect different formats — for example, a Spark-on-Databricks team reading Delta and a Trino team reading Iceberg — relying on XTable or UniForm to translate metadata is often preferable to maintaining duplicate copies of the data. The decision then shifts from “which format” to “which write path produces the data, and which translations are needed for the readers.” This framing is the clearest sign of how far the field has moved from the original format competition.

Frequently Asked Questions

What is a lakehouse table format, and how does it differ from a file format like Parquet?

A file format such as Parquet defines how the bytes of a single data file are laid out. A table format is a metadata layer above many such files that presents them as one logical table with ACID transactions, schema evolution, and time travel. Parquet stores the data; the table format records which files belong to which committed version of the table and how to read them consistently.

Is Apache Iceberg replacing Delta Lake and Hudi?

Not exactly. Iceberg has become the de-facto common denominator that the other formats translate toward, and it is the safe default for new neutral deployments. Delta Lake and Hudi retain distinct strengths — deepest Spark integration and the most mature streaming-upsert tooling, respectively — and both can now expose their tables as Iceberg, so they continue to be used as write paths even where Iceberg is the serving format.

What is the difference between Hudi Copy-on-Write and Merge-on-Read?

Copy-on-Write rewrites the base file whenever a record is updated, which makes reads fast and writes heavier; it suits read-heavy tables with moderate update rates. Merge-on-Read appends updates to delta log files and merges them with the base file at query time, which makes writes light and low-latency; it suits high-frequency upserts, with background compaction periodically consolidating the deltas.

Does choosing one format lock an organization into one vendor or engine?

Less than it once did. Delta UniForm exposes Delta tables as Iceberg through the Iceberg REST catalog, Hudi can output Iceberg-compatible tables, and Apache XTable translates metadata among all three with no data copying. Iceberg in particular is vendor-neutral by design. Lock-in is now mainly a function of operational tooling and the write path rather than the storage format itself.

What did the Databricks acquisition of Tabular mean for the format landscape?

Databricks, the company behind Delta Lake, acquired Tabular — the startup founded by Iceberg’s original creators — for more than one billion dollars, with the deal revealed on June 4, 2024. Bringing Iceberg’s architects inside the Delta vendor signaled support for both formats and is widely read as the point at which the industry pivoted from a format competition toward interoperability.

References

• Apache Iceberg — official documentation
• Delta Lake — official site and documentation
• Apache Hudi — official documentation
• Apache XTable (incubating) — omni-directional table-format interoperability
• Databricks — acquisition of Tabular (announced June 4, 2024)

Comparative characterizations of engine reach and format maturity reflect 2026 guidance from RisingWave, Dremio, Onehouse, and lakeFS; figures and dated claims are attributed inline.

Conclusion

The lakehouse table format is the layer that turns object storage into a transactional database surface, and the three principal implementations express three design priorities: Iceberg’s engine-neutral interoperability, Delta Lake’s depth within the Spark and Databricks ecosystem, and Hudi’s strength in high-frequency streaming upserts. Their core guarantees converge, but their metadata models and operational characters remain distinct enough to matter on the write path.

The decisive shift of the past two years is that selecting a format no longer means accepting permanent lock-in. The Databricks–Tabular acquisition, Delta UniForm, Hudi’s Iceberg output, and Apache XTable have made it possible for one physical dataset to be presented as more than one format, with Iceberg emerging as the de-facto common denominator. The practical recommendation that follows is straightforward: default to Iceberg for new, multi-engine, vendor-neutral deployments; choose Delta Lake within Databricks-centric stacks; choose Hudi for upsert-heavy streaming ingestion; and lean on interoperability layers when one dataset must serve consumers expecting different formats. The right choice is the one that fits the write path, because interoperability can serve the readers afterward.

This article examines the architecture of Kubernetes pods and explains why directly connecting an external client to a database running inside a pod is more involved than the equivalent task with a standalone Docker container. The discussion is grounded in the networking model that Kubernetes uses and in the Service abstraction that the model requires. A common experience for engineers new to Kubernetes is that kubectl exec into a Postgres pod followed by psql -h localhost works as expected, while a parallel attempt from a developer laptop, using the pod IP reported by kubectl get pods -o wide, times out with no error. The credentials, the database, and the apparent network are the same, yet the second connection never completes. This outcome is not the result of a defect; it is a direct consequence of how the platform is designed, and understanding the design is the first step toward connecting to in-cluster databases in a reliable manner.

Why Kubernetes Exists in the First Place

Docker addressed a genuine packaging problem by allowing an application and its dependencies to be shipped as a single reproducible image that behaves consistently on a developer laptop, a continuous integration runner, and a production virtual machine. Readers who have not yet worked through the container model will find the Docker containers, from dev to production guide a useful prerequisite for the material that follows. Once an organization operates several dozen containers distributed across several dozen servers, however, Docker alone becomes insufficient. Several questions arise that the single-host model does not answer: which host should run a given container, what should happen if that host fails outside business hours, how can a new version be rolled out without dropping in-flight requests, how do containers on one host discover containers on another, and how should compute and memory budgets be enforced.

Kubernetes is the answer that became the industry consensus. It originated inside Google as a re-implementation of the Borg system and was open-sourced in 2014. Kubernetes is best understood as a cluster operating system: the operator declares the desired state of the workload, and a chain of controllers continuously reconciles the actual state of the cluster with that declaration. The unit that Kubernetes manages is not a container directly but a pod, a wrapper around one or more tightly coupled containers that share a network identity and storage. All higher-level objects, including Deployments, Services, StatefulSets, Jobs, and CronJobs, are abstractions that ultimately specify which pods should exist, where they should run, and how they should be exposed.

The control plane performs the coordination function, while worker nodes execute the workload. Each worker runs three components in its base layer. The kubelet is the agent that receives instructions from the API server and applies them to the local node. The container runtime, which is now almost always containerd or CRI-O, executes the containers themselves; Docker as a runtime was deprecated in version 1.20 and removed in version 1.24. The kube-proxy process programs the kernel iptables or IPVS rules that route Service IPs to actual pod endpoints. Pods are scheduled on top of these three components.

The Pod: Smaller Than a VM, Bigger Than a Container

A pod is the smallest deployable unit in Kubernetes. The simplest pod runs a single container, but the abstraction exists precisely because the smallest unit that an engineer sometimes needs to ship is more than one container. A common configuration places a primary application container alongside a sidecar container that handles logging, TLS termination, metric scraping, or database proxying. All containers in the same pod share a single network namespace, which means they can communicate over localhost, and they can share filesystem volumes. They are always scheduled together onto the same node, and their lifecycles are linked: they are created together, restarted together, and terminated together.

The manifest below shows the minimum specification for a pod. Three details are worth noting. The apiVersion: v1 field indicates that pods are part of the core Kubernetes API. The specification contains a single container running an Nginx image. There is no top-level restart policy, which reflects the fact that bare pods are not self-healing: when a node fails, the pod fails with it. Bare pods are therefore rarely used in production. They are useful primarily for one-off tests and as a teaching device.

# pod.yaml — the simplest possible pod
apiVersion: v1
kind: Pod
metadata:
  name: hello-pod
  labels:
    app: hello
spec:
  containers:
  - name: web
    image: nginx:1.27
    ports:
    - containerPort: 80
    resources:
      requests:
        cpu: "50m"
        memory: "64Mi"
      limits:
        cpu: "200m"
        memory: "128Mi"

The object that an operator actually deploys is a Deployment, a controller that maintains a desired number of identical pods, handles rolling updates, and recreates pods when nodes fail. The Deployment owns a ReplicaSet, which in turn owns the pods. Operators rarely reference pods directly. Instead, they reference the Deployment, and Kubernetes manages the underlying pods.

# deployment.yaml — a real workload
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  labels:
    app: api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
      - name: api
        image: ghcr.io/acme/api:1.4.2
        ports:
        - containerPort: 8000
        readinessProbe:
          httpGet: { path: /healthz, port: 8000 }
          initialDelaySeconds: 5
        livenessProbe:
          httpGet: { path: /livez, port: 8000 }
          initialDelaySeconds: 15
        env:
        - name: DB_HOST
          value: "postgres.db.svc.cluster.local"
        - name: DB_PORT
          value: "5432"

The Flat Networking Model and Its Implications

The Kubernetes networking model rests on four rules that appear simple on the surface but carry substantial implications for how traffic flows inside a cluster:

1. Every pod gets its own IP address, drawn from a cluster-wide CIDR range that does not overlap with the node IPs.
2. Pods on the same node can communicate without NAT.
3. Pods on different nodes can communicate without NAT.
4. The IP a pod sees as its own is the same IP that other pods see when they talk to it.

The fourth point carries the most weight. In a typical single-host Docker configuration, a container holds a private IP on a bridge network, and outbound traffic is translated through the host using NAT. Kubernetes deliberately avoids this arrangement. Every pod is a first-class participant in a single flat network, regardless of the physical machine that hosts it. The component that implements this property is a CNI plugin (Container Network Interface), of which several mature implementations exist, including Calico, Cilium, Flannel, Weave, the AWS VPC CNI, and the native plugin used by GKE. These plugins implement the same contract but differ in their mechanisms, which range from overlays based on VXLAN tunnels, to BGP route advertisement, to native cloud routing, to eBPF-based data planes.

This flat addressing scheme is convenient for application code, which simply connects to an IP address and proceeds, but it is demanding for operators who must reason about traffic flows. Every pod is mutually addressable inside the cluster, which means that in the absence of explicit policies, every pod is able to reach every database, every cache, and every internal API. This property becomes important in a later section, which explains why directly addressing a database pod is more fragile than it appears.

Services: How Pods Actually Find Each Other

Because pod IPs change on every restart, they cannot appear in a connection string. The Kubernetes solution to this problem is a Service, a stable virtual IP and DNS name that load-balances traffic to a set of pods identified by labels. Individual pods come and go, but the Service remains. Inside the cluster, every Service automatically receives a DNS name of the form service-name.namespace.svc.cluster.local, which is resolved by CoreDNS, the built-in DNS resolver of the cluster.

The manifest below defines a ClusterIP Service for a Postgres pod. The selector field is worth attention, because the Service matches by labels rather than by name. Any pod that carries the label app: postgres in the same namespace automatically becomes a backend.

# service.yaml — ClusterIP for Postgres
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: db
spec:
  type: ClusterIP            # default, can omit
  selector:
    app: postgres
  ports:
  - name: pg
    port: 5432              # the Service port
    targetPort: 5432        # the container port
    protocol: TCP

From any other pod inside the cluster, the command psql -h postgres.db.svc.cluster.local -p 5432 will succeed. The same command issued from a developer laptop will hang indefinitely. The next section examines the reasons for this gap in detail.

Why Direct Connections to a Database Pod Fail

The assumption that breaks down for engineers coming from a plain Docker workflow is the idea that a container can be reached as long as one knows its IP address and port. In Kubernetes, almost every element of that assumption is incorrect. The pod IP is real but private, ephemeral, and exists only on a network shared between the nodes of a single cluster. The container port is open inside the container but is not automatically exposed at any higher layer. There is no host-level port-publishing equivalent to docker run -p 5432:5432; the field hostPort exists but is discouraged for production use. The following paragraphs examine each failure mode in turn.

Pod IPs are ephemeral. The moment a pod restarts, for any reason ranging from a node reboot, to a failed liveness probe, to a manual kubectl rollout restart, to an eviction by the scheduler, the new pod receives a new IP address from the address pool that the CNI manages. Any client that retains a reference to the previous IP is now communicating with nothing, or in the worst case, with whatever pod has been allocated the recycled address. This is the reason pod IPs should never be written into a configuration file. The correct address to record is a Service DNS name, which CoreDNS resolves at lookup time.

The ClusterIP is not visible from outside the cluster. The Service IP that kubectl get svc reports, such as 10.96.42.7 in the earlier example, is a virtual IP. It does not belong to any physical or virtual network interface and exists only as an entry in the iptables tables that kube-proxy maintains on each node. A laptop outside the cluster has no route to the Service CIDR 10.96.0.0/12, and even a statically added route would not help, because no kernel outside the cluster contains the rules required to translate that virtual address.

Pods do not use the host network by default. Setting hostNetwork: true on a pod causes the container to share the network namespace of the node, with the consequence that the container port maps directly to a port on the node. This configuration is used by CNI agents, node-exporter, and similar infrastructure components. Applying it to a database, however, is poor practice: IP isolation is lost, port collisions become possible, and any node failure takes the database with it, since the address is tied to a specific host and cannot be moved.

NetworkPolicies can explicitly deny traffic. When the cluster runs a CNI that supports NetworkPolicy, which most modern plugins do, operators can write rules such as “only pods labeled role: api in the app namespace may connect to pods labeled app: postgres in the db namespace on port 5432.” When a default-deny baseline is in place and no allow rule has been written, all traffic is dropped. When no policies are present at all, all traffic is permitted, which presents its own security concerns.

# networkpolicy.yaml — only the api can talk to postgres
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: postgres-allow-api
  namespace: db
spec:
  podSelector:
    matchLabels:
      app: postgres
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: app
      podSelector:
        matchLabels:
          role: api
    ports:
    - protocol: TCP
      port: 5432

The container port is not automatically exposed at the node level. Docker users are accustomed to -p 5432:5432, which binds a host port to a container port. Kubernetes provides no equivalent automatic mapping. The containerPort field in a pod specification is documentation: it informs operators and tooling that the container intends to listen on the indicated port, but it does not open a path through any higher layer. External reachability requires a Service of the appropriate type and, in cloud environments, a security group rule that permits traffic to whichever node port the cloud load balancer or NodePort uses.

Databases are stateful, and stateful pods require stateful controllers. A plain Deployment treats its pods as interchangeable replicas. The Deployment will reschedule postgres-0 from node 2 to node 5 when node 2 becomes unhealthy, mounting whichever PersistentVolume is available, or no volume at all if the PersistentVolumeClaim has been deleted. A database instead requires a StatefulSet, which assigns each pod a stable identity such as postgres-0 or postgres-1, a stable per-pod DNS name served by a headless Service, and a stable PersistentVolumeClaim that remains attached to the same ordinal across reschedules. A misconfiguration in this area is a common cause of data loss for teams new to running databases on Kubernetes.

The request path is long, and any single weak link breaks it. When an external client reaches a pod, the request typically traverses the following sequence: client, public DNS, cloud load balancer, node IP, iptables DNAT, pod IP, container port, Postgres listener, pg_hba.conf check, and finally authentication. A misconfiguration at any stage, such as an incorrect TLS certificate, a security group blocking the load balancer health check, a pg_hba.conf rule that denies the source CIDR, or a Postgres listener bound to 127.0.0.1 inside the container rather than 0.0.0.0, produces a connection failure that appears identical to a network problem from the perspective of the client.

Three Reliable Connection Patterns

Three legitimate patterns exist for connecting a client to a database that runs in a pod, and the appropriate choice depends primarily on the location of the client. Selecting among them is largely a question of which client requires the connection and for how long.

Pattern A: In-cluster application to in-cluster database

This pattern is the default and the most reliable choice. The application pod sets DB_HOST=postgres.db.svc.cluster.local as an environment variable and opens a connection. CoreDNS resolves the name, kube-proxy translates the virtual IP into the address of a real pod through DNAT, and the connection succeeds. Pod restarts on either side remain transparent because every endpoint is named rather than pinned to a specific IP. This is also the pattern that Airflow workloads adopt when they run with the KubernetesExecutor described in the Apache Airflow data pipeline orchestration guide, in which each task is launched as a pod that reaches the database through a Service. The same pattern applies to dbt jobs running on Kubernetes and to Kafka consumer workloads running in pods.

Pattern B: Local developer to in-cluster database

The command kubectl port-forward opens a tunnel from a local port on a developer machine, through the Kubernetes API server, to a port on a pod. It is intended for development and one-off administrative tasks. The example below uses it against the headless Service that the next subsection defines:

# forward localhost:5432 to the postgres-0 pod's port 5432
kubectl port-forward -n db pod/postgres-0 5432:5432

# Or forward through the headless Service to whichever endpoint is selected
kubectl port-forward -n db svc/postgres 5432:5432

# Now from another terminal, on your laptop:
psql -h localhost -p 5432 -U app -d production

The Python client below connects through the forwarded port. The connection string specifies localhost, which is correct on the developer laptop. Inside the cluster, the same code would instead specify postgres.db.svc.cluster.local.

# dev_query.py — assumes "kubectl port-forward" is running
import os
import psycopg2
from psycopg2.extras import RealDictCursor

# Local dev: connect through kubectl port-forward
# In production (in-cluster), DB_HOST would be postgres.db.svc.cluster.local
DB_HOST = os.environ.get("DB_HOST", "localhost")
DB_PORT = int(os.environ.get("DB_PORT", "5432"))
DB_NAME = os.environ.get("DB_NAME", "production")
DB_USER = os.environ.get("DB_USER", "app")
DB_PASS = os.environ["DB_PASS"]  # required, no default

def fetch_recent_orders(limit: int = 50):
    """Read the most recent orders — example dev-time query."""
    with psycopg2.connect(
        host=DB_HOST,
        port=DB_PORT,
        dbname=DB_NAME,
        user=DB_USER,
        password=DB_PASS,
        connect_timeout=5,
        sslmode="require",   # still enforce TLS even on port-forward
    ) as conn:
        with conn.cursor(cursor_factory=RealDictCursor) as cur:
            cur.execute(
                "SELECT id, customer_id, total_cents, created_at "
                "FROM orders ORDER BY created_at DESC LIMIT %s",
                (limit,),
            )
            return cur.fetchall()

if __name__ == "__main__":
    rows = fetch_recent_orders()
    for row in rows:
        print(row)

Pattern C: External application to in-cluster database

This is the pattern about which most teams should hesitate. When an application outside the cluster needs to read from or write to the database, the preferred architecture is almost always to expose an API over HTTP or gRPC through an Ingress with TLS and authentication, and to let the API mediate access to the database. Legitimate cases for direct external access nevertheless exist, including analytics tools, business intelligence dashboards, and replication to external systems. In those cases the pattern takes the following shape: a Service of type LoadBalancer backed by the database pods, fronted by a cloud network load balancer, with the security group restricted to specific client CIDRs, mandatory TLS, and a credential rotation policy. When a managed database such as Amazon RDS, Google Cloud SQL, or Aurora can be substituted, that option is usually preferable. Operating Postgres inside Kubernetes is technically feasible, but it represents a significant operational commitment.

The StatefulSet plus headless Service pattern

A headless Service is the object produced when clusterIP: None is set in the specification. Rather than allocating a virtual IP, this configuration produces DNS A records, with one record per pod backend. When combined with a StatefulSet, the result is a set of stable per-pod hostnames, such as postgres-0.postgres.db.svc.cluster.local and postgres-1.postgres.db.svc.cluster.local. This naming arrangement is precisely what a primary-replica database deployment requires. The application directs writes to the hostname of the primary and reads to the hostname of any replica.

# headless service + statefulset for postgres
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: db
  labels:
    app: postgres
spec:
  clusterIP: None          # headless — no virtual IP
  selector:
    app: postgres
  ports:
  - name: pg
    port: 5432
    targetPort: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: db
spec:
  serviceName: postgres    # MUST match the headless Service name
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      terminationGracePeriodSeconds: 30
      containers:
      - name: postgres
        image: postgres:16.3
        ports:
        - containerPort: 5432
          name: pg
        env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: password
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
        readinessProbe:
          exec:
            command: ["pg_isready", "-U", "postgres"]
          initialDelaySeconds: 10
          periodSeconds: 5
        resources:
          requests:
            cpu: "500m"
            memory: "1Gi"
          limits:
            cpu: "2"
            memory: "4Gi"
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: gp3-ssd
      resources:
        requests:
          storage: 200Gi

Production databases almost always benefit from a purpose-built operator layered on top of this scaffolding, such as CloudNativePG, the postgres-operator developed by Zalando, or Crunchy PGO. These operators handle primary election, streaming replication, backups, point-in-time recovery, and rolling minor-version upgrades. Selecting an appropriate database backend is a separate concern; the database comparison for preprocessed time-series data serves as a useful companion reference for that decision.

Kubernetes 1.36 and What Changed in 2026

Kubernetes 1.36 is the most recent minor release as of this writing in May 2026, and it continues the project’s emphasis on stronger security defaults and on first-class support for AI workloads. According to the official release page (Source: kubernetes.io/releases, as of 2026-05-20), the project actively maintains release branches for the three most recent minor versions, currently 1.34, 1.35, and 1.36. Version 1.33 entered maintenance on 2026-04-28 and reaches end of life on 2026-06-28. The release cadence is rapid enough that operators running anything older than 1.33 are already outside the supported window.

The promotion of User Namespaces to general availability is the most prominent security change in 1.36. When user namespaces are enabled, the root user inside a container is mapped to an unprivileged user on the host. This arrangement substantially reduces the impact of a container escape: even when an attacker compromises a container running as UID 0, they emerge on the host as a high-numbered unprivileged user, such as UID 100000, with no special privileges. For database pods specifically, a compromised Postgres container no longer translates directly into root access on the node. In combination with seccomp and AppArmor profiles, this change closes one of the long-standing gaps between Kubernetes security and traditional virtual machine isolation.

Mutating Admission Policies, also promoted to general availability, bring declarative mutations expressed in the Common Expression Language (CEL) to the admission chain, replacing many uses of webhook-based mutating admission controllers. Operators can now write policies that, for example, automatically inject sidecar containers, attach labels, set default resource requests, or enforce image-registry rules, without operating a separate webhook server. The result is less infrastructure to maintain and fewer failure modes when a webhook becomes unavailable.

Fine-Grained Kubelet API Authorization, now generally available, allows the kubelet to enforce per-verb RBAC on its own API rather than treating all operations uniformly. This change matters for hardening: tools that require nodes/proxy can be restricted to read-only operations, and the kubelet can refuse risky combinations that previously required cluster-admin privileges in order to be fully restricted.

Beyond security, version 1.36 continues to invest in AI workload support. It introduces refinements to Dynamic Resource Allocation (DRA) for GPU scheduling, adds support for accelerator partitioning, and improves the ability of the scheduler to handle long-running training jobs alongside short-lived inference pods. The trajectory is clear: the pattern of Kubernetes as an AI platform, which grew rapidly in 2024 and 2025 as model-serving workloads migrated off bespoke infrastructure, has been a first-class concern for two consecutive release cycles. For language and runtime choices when developing operators or controllers around these new APIs, the Python and Rust comparison provides a useful framing. The controller-runtime ecosystem in Go remains dominant, but Rust-based operators are gaining ground for performance-sensitive components.

Frequently Asked Questions

Can a pod have more than one container?

Yes, and it is a common design pattern. The most frequent reason is the sidecar — a helper container that does logging, TLS termination, service-mesh proxying (Envoy in Istio or Linkerd), or connection pooling. All containers in a pod share a network namespace and can share volumes, but they remain separate processes with separate filesystems. Use multiple containers when their lifecycles are genuinely coupled. If the answer to “can these scale independently?” is yes, they belong in separate pods.

Why not just expose every database pod with a NodePort and connect directly?

NodePort opens the same port on every node in the cluster, in the 30000–32767 range, and routes it to whichever pod backs the Service. Three problems: the port numbers are non-standard so client tooling fights you, every node becomes an attack surface for the database, and you still need a cloud security group or firewall rule to control who can hit those ports. NodePort is fine for on-prem clusters without a cloud LB or for very specific debug scenarios. It is not a substitute for proper Service architecture.

Is kubectl port-forward safe to use in production?

It is safe to use, but it should not be how production traffic flows. The tunnel runs through the API server and consumes API-server resources. It bypasses NetworkPolicy — if you can port-forward, you can connect, regardless of how strict your in-cluster policies are. RBAC controls who can use it, and you should treat pods/portforward on a database namespace as a sensitive verb subject to audit. For production traffic, use a real Service.

What is the difference between a StatefulSet and a Deployment?

A Deployment treats pods as interchangeable. It will scale up by spinning up new pods with random suffix names, scale down by killing any of them, and roll updates in parallel. A StatefulSet maintains ordered, named pods (name-0, name-1, name-2) that always come up in order, always shut down in reverse order, and each get their own stable PersistentVolumeClaim. Use Deployment for stateless apps. Use StatefulSet for anything that has identity — databases, message brokers, ZooKeeper, distributed coordination services. Kafka brokers running in Kubernetes are a textbook StatefulSet workload.

Should I actually run my database in Kubernetes, or use a managed service?

For most teams below the scale of needing a database engineer on the org chart, managed (RDS, Cloud SQL, Aurora, AlloyDB, Spanner) is the right answer. Operating a stateful workload well — backups, point-in-time recovery, minor-version upgrades, failover, performance tuning, observability — is a continuous engineering investment that managed services amortize across thousands of customers. Run databases in your cluster when you have a real reason: cost at scale, regulatory data residency, latency requirements that make a separate database tier unworkable, or a database that managed offerings do not provide. The operator ecosystem (CloudNativePG and friends) makes this much more tractable than it was five years ago, but it is still real work.

Related Reading

References

• Kubernetes releases — official release and support timeline
• Kubernetes cluster networking concepts
• Kubernetes 1.36 sneak peek (official blog, March 2026)
• Connecting applications with Services (official tutorial)
• InfoQ: Kubernetes 1.36 released (May 2026)

Conclusion

Connecting to a database that runs in a Kubernetes pod feels harder than it should because Kubernetes is solving a different problem than many engineers initially assume. It is not an elaborate replacement for docker run. It is a cluster operating system whose entire networking model is designed around the principle that pods communicate with other pods through stable abstractions, and external clients reach applications through carefully chosen entry points. The pod IP revealed by kubectl get pods -o wide is a debugging convenience rather than an address suitable for client traffic. The ClusterIP shown by kubectl get svc is a virtual construct held together by iptables rules. The correct address for production traffic originating inside the cluster is a DNS name served by CoreDNS and backed by a Service whose membership the controllers maintain. The correct address from outside the cluster is whatever the LoadBalancer, Ingress, or bastion-host configuration specifies, and it is never a pod IP.

Three points are worth retaining from this discussion. First, kubectl port-forward is well suited to development workflows and unsuited to production traffic. Second, stateful workloads require a StatefulSet, a PersistentVolumeClaim, and a headless Service in combination, or data loss is likely. Third, in Kubernetes 1.36 and beyond, security defaults are tightening, with User Namespaces reaching general availability as the most consequential change, which benefits anyone running databases in pods. Even with these improvements, however, the number of ways in which a connection between an external client and an in-cluster database can fail remains large enough that exposing Postgres directly to the public internet is almost always inferior to placing an API tier in front of the database. The recommended approach is to build the conservative, layered version first, and to reserve more aggressive shortcuts for cases that genuinely warrant them.