Author: kongastral

Why codec choice is a size-versus-speed decision

Storing the same 10,000,000-row table as Apache Parquet produced files ranging from 125.06 MiB with no compression down to 68.52 MiB with Zstandard at level 9 — a difference of roughly 45 percent in on-disk footprint from a single writer setting. The write cost, however, ranged from 0.7139 seconds to more than 44 seconds depending on the codec chosen. Those two numbers, measured on one machine over one dataset, frame the practical question this benchmark addresses: which Parquet compression codec to select, and why the answer is rarely the codec that produces the smallest file.

Apache Parquet is a columnar file format, meaning values from the same column are stored together rather than interleaved row by row. Compression is applied per column chunk after encoding, so the codec operates on runs of similar values, which is why columnar layout compresses far better than row-oriented storage. The format specification defines a fixed set of codecs — among them SNAPPY, GZIP, ZSTD, and LZ4 — and each represents a different point on the curve that trades computation for space. The mechanics of columnar layout, page encoding, and predicate pushdown are covered in a companion article on Apache Parquet and Apache Arrow internals; the focus here is narrower and empirical, measuring what those codecs actually cost and save on one representative dataset.

The reason a benchmark is worth running rather than reasoning from first principles is that the tradeoffs interact. A codec that compresses tightly can be slow enough to write that it stalls an ingestion pipeline. A codec that decompresses quickly may barely shrink the file. Row-group size, which controls how finely a query engine can skip data, changes compression ratio at the same time. Dictionary encoding, applied before the codec runs, can matter more than the codec itself for certain columns. Measuring these effects together, on the same data with the same tooling, is the only way to see which choices dominate.

Test environment: hardware, software, and dataset

Compression benchmarks are only interpretable when the machine, the library versions, and the data are stated exactly, because decompression speed is CPU-bound and codec implementations differ between library releases. The complete environment capture is recorded in the benchmark repository; the essentials are reproduced below.

Hardware and software

All writing and reading was done through the PyArrow Parquet writer, so the results reflect the Arrow C++ implementations of each codec at version 25.0.0. A different language binding or a different Arrow release could shift the absolute times, which is precisely why the versions are pinned and stated. The PyArrow write_table documentation lists the codec names accepted by this writer and confirms that its default codec is snappy.

The dataset

The benchmark reused a single 10,000,000-row table shared with a sibling study on DuckDB and Polars in-process analytics, so that both studies run on identical data. The table has five columns of mixed type, an in-memory Arrow footprint of 312,390,663 bytes (297.9 MiB), and the following schema:

• ts — timestamp[us], event time spanning roughly 90 days (November 2023 to February 2024).
• user_id — int32, values in the range 1 to 499,997.
• category — string, low cardinality with 10 distinct values (books, automotive, electronics, garden, beauty, apparel, home, grocery, and others).
• amount — float64, values from 0.09 to 3,714.81.
• flag — bool.

Two preparation steps were applied once, before any timing. First, the category column was decoded from its stored dictionary form to plain UTF-8 text, so that the writer’s dictionary flag — and not a carried-over Arrow dictionary — is the only thing deciding whether dictionary encoding is applied at the file level. Second, the table was sorted by ts ascending. Time-series data is realistically stored in time order, and sorting makes each row group hold a tight, non-overlapping range of timestamps, which is a precondition for the predicate-pushdown measurements described later.

Methodology: how every number was produced

Each measurement was repeated three times, and the median of the three runs is quoted throughout; the recorded results file also stores the full list of run times for inspection. Timings use a wall-clock performance counter, and garbage collection is forced before each timed operation to reduce interference.

One property of the setup deserves explicit statement rather than silent assumption: the read-cache state is warm. Reads happen immediately after each file is written, so the file is present in the operating-system page cache. Read timings therefore measure the CPU cost of decoding and parsing the columnar data, not cold disk input. This is the standard basis for comparing codecs because it isolates decompression speed from storage latency, but it means the read numbers should be read as decode cost, not as the end-to-end latency of a query against a cold object store. File sizes, by contrast, are exact on-disk byte counts, and per-column compressed and uncompressed sizes are read directly from the Parquet footer metadata.

Three separate experiments were run. The codec matrix fixed the row-group size at 1,000,000 rows and dictionary encoding on, then wrote and read the table with each of six codecs. The row-group study fixed the codec at Zstandard level 3 and varied the row-group size. The dictionary study fixed the codec at Zstandard level 3 and wrote the table twice, once with dictionary encoding on and once off, comparing the string column. The raw console output of the codec matrix is reproduced verbatim below.

==============================================================================
CODEC MATRIX  (row_group_size=1000000, dictionary encoding ON)
==============================================================================
predicate: ts >= 2024-02-08 10:13:18.975015  (expected ~498,711 rows, 4.99%)

[none    ] size=  125.06 MiB  write_med=0.7139s  read_med=0.0739s  col_med=0.0083s  filt_push_med=0.0176s (rows=498,711)  filt_nopush_med=0.0163s
[snappy  ] size=  102.17 MiB  write_med=0.8760s  read_med=0.0890s  col_med=0.0092s  filt_push_med=0.0170s (rows=498,711)  filt_nopush_med=0.0303s
[zstd_l3 ] size=   68.61 MiB  write_med=0.9421s  read_med=0.0783s  col_med=0.0140s  filt_push_med=0.0173s (rows=498,711)  filt_nopush_med=0.0406s
[zstd_l9 ] size=   68.52 MiB  write_med=2.3837s  read_med=0.0755s  col_med=0.0137s  filt_push_med=0.0171s (rows=498,711)  filt_nopush_med=0.0407s
[gzip    ] size=   74.90 MiB  write_med=44.3629s  read_med=0.1048s  col_med=0.0352s  filt_push_med=0.0230s (rows=498,711)  filt_nopush_med=0.0900s
[lz4     ] size=  102.65 MiB  write_med=0.8580s  read_med=0.0858s  col_med=0.0084s  filt_push_med=0.0172s (rows=498,711)  filt_nopush_med=0.0216s

The codec matrix: file size, write speed, read speed

The full codec matrix is shown in the table below. Write and read medians are in seconds; file size is the exact on-disk footprint in MiB; the single-column read measures reading only the amount column.

Three findings stand out. First, the two Zstandard settings produced almost the same file — 68.61 MiB at level 3 versus 68.52 MiB at level 9 — a difference of under 0.10 MiB, yet level 9 took 2.3837 s to write against 0.9421 s for level 3, roughly two and a half times longer. On this data, the higher level buys essentially nothing for a substantial write cost. Second, gzip is dominated on every axis that matters: its 74.90 MiB file is larger than either Zstandard result, and its 44.3629 s median write is about 47 times slower than Zstandard level 3. gzip is the only codec whose write time is measured in tens of seconds rather than fractions of a second.

Third, snappy and lz4 behave almost identically here: 102.17 MiB against 102.65 MiB, with writes of 0.8760 s and 0.8580 s respectively. Both are fast, lightly compressing codecs that shrink the file to about 82 percent of the uncompressed size. snappy is the writer’s default, and neither of these two clearly beats the other on this dataset. It is worth noting that the Parquet specification marks the older LZ4 codec as deprecated in favor of a newer LZ4_RAW framing; the codec exercised here is the one PyArrow writes under the lz4 name.

Read performance tells a flatter story. On a warm page cache, decoding the full table took between 0.0739 s (no compression) and 0.1048 s (gzip). Zstandard sat between the extremes at 0.0783 s for level 3 and 0.0755 s for level 9, and snappy and lz4 landed at 0.0890 s and 0.0858 s. In other words, choosing a strong general-purpose codec added only a few milliseconds to a full decode of the whole table relative to no compression at all, while gzip again stood out as the slowest to read. When the read touched only the single amount column, the absolute times dropped by roughly an order of magnitude, but the same ordering held, with gzip slowest at 0.0352 s.

Filtered reads and the price of decompression

Parquet stores per-row-group statistics, including the minimum and maximum of each column, which lets a reader skip entire row groups whose range cannot satisfy a predicate. This is predicate pushdown, and its benefit depends on the data being sorted so that ranges do not overlap. Because the table was sorted by ts, a predicate of ts >= a late cutoff — matching 498,711 rows, or 4.99 percent of the table — allowed the reader to skip most row groups.

The benchmark measured two versions of that filtered read. The pushdown version passed the predicate to the reader, which used row-group statistics to skip groups. The control version read the relevant columns in full and filtered them in memory afterward, doing the work that pushdown avoids. The contrast is clearest for the compressed codecs. For Zstandard level 3, the pushdown read took 0.0173 s while the no-pushdown control took 0.0406 s; for snappy the two were 0.0170 s and 0.0303 s. Skipping row groups avoids decompressing them, so the saving grows with how expensive the codec is to decode.

The uncompressed case is the instructive exception. With no compression, the pushdown read (0.0176 s) was marginally slower than the no-pushdown control (0.0163 s), because there is no decompression to avoid and the pushdown path carries a small bookkeeping overhead. In short, predicate pushdown and compression reinforce each other: the more a codec costs to decode, the more a query gains by skipping data it never needed to read. This interaction is central to why lakehouse query engines lean heavily on statistics-based skipping over columnar files.

Row-group size: compression against skipping granularity

A row group is the horizontal partition of a Parquet file within which columns are chunked and compressed. Its size controls two things at once. Larger row groups give the codec more context and amortize per-group overhead, which improves compression; smaller row groups create more, finer boundaries, which lets predicate pushdown skip data more precisely. These pull in opposite directions, and the benchmark measured the tension directly by fixing the codec at Zstandard level 3 and varying the row-group size across 131,072 rows, 1,000,000 rows, and 5,000,000 rows.

The compression trend is monotonic: the file shrank from 87.41 MiB at 131,072 rows per group to 68.61 MiB at one million and 66.79 MiB at five million. Small row groups pay a real penalty, here roughly 20 MiB, because encoding structures such as the dictionary reset at every group boundary and the codec sees less data per chunk. The selective-read trend runs the other way. With 77 fine-grained groups, the pushdown filter completed in 0.0070 s, because the reader could discard almost every group; with only two coarse groups it took 0.0853 s, because a matching predicate forces reading a full half of the table. The full-scan read was fastest at the smallest row-group size (0.0587 s) and slowest at the largest (0.1922 s) in this run, though full-scan timing is more sensitive to how decode work parallelizes across groups and should be treated with more caution than the size and pushdown figures.

Dictionary encoding on a low-cardinality column

Dictionary encoding replaces repeated column values with small integer indices into a per-column dictionary, and it is applied before the compression codec runs. For a column with few distinct values it can shrink the data dramatically on its own, which changes how much work is left for the codec. The category column is a natural test: 10 distinct string values repeated across 10,000,000 rows. Holding the codec at Zstandard level 3, the table was written once with dictionary encoding on and once with it off.

The effect was large. With dictionary encoding on, the category column occupied 3.912 MiB compressed on disk, having been reduced to 4.813 MiB uncompressed before the codec even ran. With dictionary encoding off, the same column expanded to 104.825 MiB uncompressed — the raw repeated strings — and although Zstandard still compressed that down, it landed at 14.172 MiB, about 3.6 times the size of the dictionary-encoded column. The whole-file impact followed: the file grew from 68.61 MiB with the dictionary to 87.40 MiB without it, an increase of 18.79 MiB attributable to a single column.

The practical lesson is that for low-cardinality columns, dictionary encoding can matter more than the choice of codec. A strong general-purpose codec applied to raw repeated strings did not recover what dictionary encoding achieved almost for free. PyArrow enables dictionary encoding by default, and the Parquet specification defines the modern RLE_DICTIONARY encoding for exactly this pattern. The failure mode to watch for is a high-cardinality column where the dictionary grows too large to help; the format falls back to plain encoding in that case, and forcing dictionary encoding on such a column wastes effort. The behavior is well matched to the kind of categorical fields common in batch and streaming data pipelines.

Choosing a codec: practical guidance

The measurements support a short, honest set of recommendations for this class of tabular, analytics-oriented data. They are grounded in one dataset on one machine, so they are guidance rather than universal law, but the gaps between codecs are large enough to be robust to modest variation.

Zstandard at a low level is the strongest default. Level 3 produced the smallest practical file (68.61 MiB) at a write cost (0.9421 s) close to the fast codecs and a read cost (0.0783 s) within a few milliseconds of no compression at all. It is the setting that most cleanly balances the three axes. Raising the level to 9 is not worth it on data like this, where it saved under 0.10 MiB for more than double the write time; higher Zstandard levels earn their cost only when the data compresses much further and storage or transfer dominates.

snappy or lz4 make sense when write throughput and decode latency matter more than storage, and the data is written far more often than it is read. Both wrote in about 0.86 to 0.88 s and produced files near 102 MiB, meaning they leave roughly a third of the compressible space on the table in exchange for speed and simplicity. snappy being the PyArrow default is a reasonable choice for intermediate or short-lived files. gzip has no winning case in these results: it was both larger and far slower to write than Zstandard, so Zstandard should be preferred wherever gzip might have been chosen for ratio. Leaving data uncompressed is defensible only for very short-lived scratch files where even a sub-second write matters and the 125.06 MiB footprint is irrelevant.

Two settings beyond the codec deserve equal attention. Row-group size should be tuned to the query pattern: around one million rows is a sound default for time-sorted data read by range, capturing most of the compression benefit while keeping selective reads fast. Dictionary encoding should be left on for low-cardinality columns, where it delivered a larger saving than the codec choice itself. These file-level decisions carry across the broader storage stack, including the open table formats that manage collections of Parquet files, and they compound with engine-level execution choices such as those examined in the article on Apache Spark internals.

Frequently Asked Questions

Which Parquet compression codec is the best default?

On this benchmark, Zstandard at level 3 was the strongest all-round default: it produced the smallest practical file at 68.61 MiB while writing in 0.9421 s and reading in 0.0783 s, within a few milliseconds of no compression. snappy, which PyArrow uses by default, is a reasonable alternative when write speed matters more than storage, but it left the file about 50 percent larger at 102.17 MiB.

Is a higher Zstandard compression level worth the cost?

Not on data like this. Moving from level 3 to level 9 reduced the file by under 0.10 MiB, from 68.61 MiB to 68.52 MiB, while write time rose from 0.9421 s to 2.3837 s, roughly two and a half times longer. Higher levels earn their cost only when the data compresses substantially further and storage or network transfer, rather than write throughput, is the binding constraint.

Why was gzip so much slower than the other codecs?

gzip recorded a median write time of 44.3629 s, about 47 times the 0.9421 s of Zstandard level 3, and it also produced a larger file (74.90 MiB versus 68.61 MiB) and the slowest full read (0.1048 s). In these results gzip was dominated on every axis, so Zstandard is the better choice wherever gzip might have been selected for compression ratio.

How does row-group size affect compression and query speed?

Smaller row groups compress worse but skip data more finely. At 131,072 rows per group the file was 87.41 MiB but a selective pushdown read took only 0.0070 s; at 5,000,000 rows per group the file shrank to 66.79 MiB but the same read rose to 0.0853 s. Around one million rows balanced the two, giving 68.61 MiB and a 0.0176 s selective read.

References

• Benchmark source, methodology, environment capture, and recorded results: jkc4416/analytics-bench (parquet-codecs study).
• Apache Parquet format, compression codecs: parquet.apache.org — Compression.
• Apache Parquet format, dictionary and RLE encodings: parquet.apache.org — Encodings.
• PyArrow writer parameters (compression, compression_level, use_dictionary, row_group_size): arrow.apache.org — pyarrow.parquet.write_table.
• Zstandard compression library and levels: facebook/zstd.

Conclusion

Across one 10,000,000-row table measured with a fixed, recorded methodology, the codec that produced the smallest file was not the one worth choosing for most workloads. Zstandard at level 3 sat at the balanced point: 68.61 MiB on disk, a 0.9421 s write, and a read cost within a few milliseconds of no compression. Level 9 spent more than twice the write time to save under a tenth of a megabyte, and gzip was both larger and dramatically slower, with a 44.3629 s median write. Two file-level settings mattered as much as the codec — a moderate row-group size preserved most of the compression benefit while keeping selective reads fast, and dictionary encoding on a single low-cardinality column changed the whole-file size by 18.79 MiB. The durable conclusion is that Parquet compression is a system of interacting choices, and the right decision comes from measuring the levers in order of impact rather than optimizing any one of them in isolation.

In an event-driven system built on Apache Kafka, every message that crosses the network is a sequence of raw bytes with no built-in description of its structure. A consumer that reads those bytes must already know how to interpret them: which fields are present, in what order, and with what types. When the producer and consumer are developed and deployed independently, and when the shape of the data changes over months of feature work, this implicit agreement becomes fragile. A schema registry addresses that fragility by turning the implicit agreement into an explicit, versioned, and centrally governed contract. This article examines how a schema registry works, how the Avro and Protocol Buffers serialization formats encode and evolve data, and how compatibility modes let a schema change without breaking the producers and consumers that already depend on it.

The discussion assumes familiarity with Kafka as a distributed log of immutable records. The registry pattern described here follows the Confluent Schema Registry, the most widely deployed implementation, but the underlying concepts, particularly Avro schema resolution and Protobuf field-number stability, are properties of the serialization formats themselves and apply wherever those formats are used.

Why Schemas Matter in Event-Driven Pipelines

A schema is a formal description of the structure of a record: the names of its fields, their data types, and rules such as whether a field may be absent. In a monolithic application, this description lives in the type system of a single codebase, and the compiler enforces it. In a distributed streaming pipeline, the producer and the consumer are separate programs, often written in different languages and released on different schedules, connected only by a stream of bytes. Nothing in the bytes themselves states what they mean.

This is the defining property of a Kafka message: it is not self-describing. A message is a key and a value, each an opaque byte array as far as the broker is concerned. The broker never parses the payload. Interpretation is entirely the responsibility of the client that deserializes it. If a producer begins writing an extra field, or renames one, or changes an integer to a string, a consumer that was compiled against the older structure has no way to detect the change from the bytes alone. It will either read garbage or fail.

The contrast with a columnar file format such as Apache Parquet is instructive. A Parquet file stores its schema in its own footer, so any reader can open the file and discover its structure without external coordination; the file is self-describing at rest. A detailed treatment of that design appears in the discussion of Parquet and Apache Arrow columnar storage internals. Kafka messages have no such footer. Attaching a full schema to every message would be prohibitively wasteful, because a stream may carry millions of records that share one structure. The registry resolves this tension by storing each schema once, assigning it a compact numeric identifier, and letting every message carry only that identifier.

This mechanism is the technical enforcement layer beneath a broader organizational concern known as a data contract, a documented and testable agreement about the meaning and quality of a dataset shared across teams. The relationship is direct: a data contract states what the data should look like, and the schema registry is one place where that statement is enforced automatically on every message. The organizational and policy dimension is treated separately in the discussion of data contracts and data quality enforcement; the present article concentrates on the wire-level mechanics. The registry also sits within a larger architectural decision about how data moves, examined in the comparison of streaming versus batch processing architectures.

Serialization Formats: Avro, Protobuf, and JSON Schema

Serialization is the process of converting an in-memory object into a byte sequence suitable for transmission or storage; deserialization is the reverse. The Confluent Schema Registry supports three serialization formats out of the box, each paired with its own serializer and deserializer: Apache Avro through KafkaAvroSerializer and KafkaAvroDeserializer, Protocol Buffers through KafkaProtobufSerializer and KafkaProtobufDeserializer, and JSON Schema through KafkaJsonSchemaSerializer and KafkaJsonSchemaDeserializer (Confluent SerDes documentation, as of 2026-07-17). The choice among them affects encoding size, tooling, and the exact rules that govern how a schema may change.

Apache Avro

Apache Avro is a data serialization system in which the schema is expressed as JSON and the data is encoded in a compact binary form that contains no field names or type tags. Because the encoding omits field identifiers, a decoder cannot interpret the bytes without a schema. Avro’s central design idea is that the schema used to write the data (the writer’s schema) and the schema used to read it (the reader’s schema) may differ, and a well-defined resolution procedure reconciles the two. The latest stable release is Avro 1.12.1, published on 2025-10-16 with four security fixes, following 1.12.0 from 2024-08-05 (Apache Avro release announcement, as of 2026-07-17).

A minimal Avro schema for an order event is a JSON object of type record:

{
  "type": "record",
  "name": "OrderPlaced",
  "namespace": "com.example.orders",
  "fields": [
    { "name": "order_id",   "type": "string" },
    { "name": "customer_id","type": "string" },
    { "name": "amount",     "type": "double" },
    { "name": "currency",   "type": "string", "default": "USD" }
  ]
}

The default on the currency field is not merely a convenience. As the compatibility section will show, a default value is the single most important element that determines whether a field can be added or removed without breaking readers. An optional field in Avro is expressed as a union with null together with a default, for example "type": ["null", "string"], "default": null. This is a distinct mechanism from optionality in Protobuf and the two should not be conflated.

Protocol Buffers

Protocol Buffers, commonly abbreviated Protobuf, is a serialization format developed by Google in which the structure is declared in a .proto file and each field is assigned a numeric tag known as a field number. In the binary encoding, the field number, not the field name, identifies each value. The current major syntax is proto3; an emerging direction called editions is intended to supersede the proto2 and proto3 distinction, but there is no single version number that meaningfully anchors the format. The same order event expressed in proto3 is:

syntax = "proto3";
package com.example.orders;

message OrderPlaced {
  string order_id    = 1;
  string customer_id = 2;
  double amount      = 3;
  string currency    = 4;
}

The numbers 1 through 4 are the field numbers. They are the anchor of Protobuf compatibility. Once a field number is assigned and data has been written with it, that number must never be changed and must never be reused for a different field, a rule examined in detail in the compatibility section (protobuf.dev best practices, as of 2026-07-17).

JSON Schema and a comparison

JSON Schema describes the structure of JSON documents. Its principal advantage is human readability and native compatibility with systems that already exchange JSON. Its principal cost is size and speed: JSON is a text encoding that repeats every field name in every message and requires parsing text rather than reading a compact binary layout. The three formats occupy different points on a tradeoff surface.

No format is uniformly superior. Avro is common in data-lake and analytics pipelines where its compact encoding and rich resolution rules fit batch and streaming loads; systems that emit change events, such as those built with Debezium, frequently produce Avro records backed by a registry, as described in the treatment of change data capture with Debezium and Kafka. Protobuf is common where the same message types are also used in remote procedure calls and where code generation across many languages is a priority. JSON Schema suits teams that value readability and already operate on JSON. The registry supports all three uniformly.

Schema Registry Architecture and the Wire Format

The Confluent Schema Registry is a service that stores schemas and assigns each a globally unique numeric identifier. It organizes schemas under subjects. A subject is a named scope under which a sequence of schema versions accumulates; each registration under a subject produces a new version number, and each distinct schema receives a global schema ID that is unique across the whole registry. Compatibility checks are performed within a subject. Producers and consumers interact with the registry over a REST API and cache results locally to avoid a lookup on every message.

Subject naming strategies

The name of the subject under which a schema is registered is determined by a subject naming strategy. Three strategies are provided (Confluent SerDes documentation, as of 2026-07-17):

• TopicNameStrategy (the default): the subject is <topic>-key or <topic>-value. This requires every message in a topic to conform to a single schema, which is the most common arrangement.
• RecordNameStrategy: the subject is the fully qualified record name. This permits several record types to share one topic, and compatibility for that record name is scoped across every topic that carries it.
• TopicRecordNameStrategy: the subject is <topic>-<fully qualified record name>. This also permits multiple record types per topic, but scopes compatibility per record type within each topic.

The 5-byte wire format

When a serializer writes a message, it does not embed the schema. It embeds a reference to the schema in a fixed header prepended to the payload. The Confluent wire format begins each serialized value with a 5-byte header: byte 0 is a magic byte with the value 0x00, and bytes 1 through 4 hold the schema ID as a 4-byte signed 32-bit integer in big-endian order, also called network byte order. The serialized payload follows from byte 5 onward (Confluent SerDes documentation; wire-format community references, as of 2026-07-17). A consumer validates the framing by confirming that the message is longer than 5 bytes and that byte 0 equals 0x00.

The wire format is deliberately stable. Confluent states that it will not change without significant warning across multiple major releases, and that within the version identified by the magic byte it never changes in a way that would break existing readers (Confluent SerDes documentation, as of 2026-07-17). The Protobuf variant is the one exception to the simple layout: because a single .proto file may declare several message types, a Protobuf message inserts a message-index array, a length-prefixed list of variable-length integer indexes that identifies which message type was serialized, between the schema ID and the payload. Avro and JSON Schema have no such segment.

The producer and consumer flow

The registry interaction is straightforward once the framing is understood. On the producing side, the serializer extracts the schema from the record, registers it under the appropriate subject (or looks up its ID if already registered), prepends the 5-byte header, and sends the message. On the consuming side, the deserializer reads the header, extracts the schema ID, fetches the corresponding schema from the registry, and uses it, together with the consumer’s own reader schema, to decode the payload. Both sides cache aggressively, so the registry is consulted at most once per distinct schema rather than once per message.

A Python producer using the confluent-kafka library expresses this flow directly. The AvroSerializer holds a reference to the registry client, and the SerializingProducer applies it to every value before sending:

from confluent_kafka import SerializingProducer
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroSerializer

schema_str = open("order_placed.avsc").read()

registry = SchemaRegistryClient({"url": "http://schema-registry:8081"})
avro_serializer = AvroSerializer(registry, schema_str)

producer = SerializingProducer({
    "bootstrap.servers": "broker:9092",
    "value.serializer": avro_serializer,
})

order = {
    "order_id": "ORD-91",
    "customer_id": "CUST-4471",
    "amount": 42.50,
    "currency": "USD",
}
producer.produce(topic="orders", value=order)
producer.flush()

The consumer side mirrors this arrangement with a DeserializingConsumer and an AvroDeserializer. The deserialization step is where a registry-backed stream connects to ordinary consumer code; the surrounding consumer plumbing, including offset management and the poll loop, is treated in the walkthrough of an Apache Kafka consumer implementation in Python. The registry itself can be queried directly over its REST API, which is useful in scripts and continuous-integration checks. Retrieving the current compatibility mode for a subject, for example, is a single request:

# Read the compatibility mode configured for a subject
curl -s http://schema-registry:8081/config/orders-value

# Set the subject to FULL compatibility
curl -s -X PUT http://schema-registry:8081/config/orders-value \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"compatibility": "FULL"}'

# List the versions registered under a subject
curl -s http://schema-registry:8081/subjects/orders-value/versions

Compatibility Modes and the Rules of Safe Evolution

Compatibility is the property that a schema change does not break the programs that already read or write data under the older schema. The registry enforces this by rejecting a schema registration that would violate the configured compatibility mode. This point deserves emphasis: the check happens at registration time, when a producer or a deployment pipeline attempts to register a new schema, not when an individual message is decoded. A rejected registration is the safeguard that prevents an incompatible change from ever being written to the log.

Two directions of compatibility are defined. Backward compatibility means new code can read data written by old code; a new reader accepts old data. Forward compatibility means old code can read data written by new code; an old reader accepts new data. Full compatibility requires both directions. Each of these has a transitive variant. A non-transitive mode checks a candidate schema only against the latest registered version, whereas a transitive mode checks it against every previously registered version (Confluent schema-evolution documentation, as of 2026-07-17).

Source: Confluent “Schema Evolution and Compatibility” documentation, as of 2026-07-17. A useful mnemonic follows directly from the definitions: BACKWARD means a new reader reads old data, so consumers are upgraded first; FORWARD means an old reader reads new data, so producers are upgraded first; FULL supports both directions, so the two sides may be upgraded independently.

Avro schema resolution

Avro’s compatibility behavior follows directly from its schema resolution rules, the procedure by which a reader reconciles its own schema with the writer’s schema field by field. The specification defines the outcomes precisely (Apache Avro Specification, Schema Resolution, as of 2026-07-17):

• If the writer’s record contains a field that the reader’s schema does not declare, the writer’s value for that field is ignored. This is what makes it safe for a producer to add a field that older consumers have not yet learned about.
• If the reader’s schema declares a field with a default value and the writer’s schema lacks that field, the reader supplies the default. This is what makes it safe to add a field to the reader.
• If the reader’s schema declares a field with no default and the writer’s schema lacks it, resolution fails with an error. This is the case a compatibility check is designed to prevent.
• Aliases allow a named type or field to declare alternative names, so a reader can match a writer’s old field or record name to a renamed one, which enables compatible renaming.
• Resolution recurses into arrays, maps, and unions, applying the same rules to item, value, and branch schemas.

Protobuf field-number evolution

Protobuf takes a different path to the same goal. Because the binary encoding identifies each value by its field number rather than its name, a field can be renamed freely without affecting the wire format, and a parser that encounters a field number it does not recognize simply skips it. This skipping behavior is the mechanism that makes additive evolution safe in proto3: adding a new field is compatible because older parsers ignore the unknown number, and reading data written by newer code succeeds because the extra fields are passed over (protobuf.dev, as of 2026-07-17).

The corresponding hazard is field-number reuse. Once a field number has been used, it must never be assigned to a different field, even after the original field is deleted. If a number is reused with a different type, a decoder holding old data will interpret the new field’s bytes according to the old field’s type and silently produce corrupt values. The remedy defined by the format is the reserved declaration, which permanently blocks a field number and name from being reused:

message OrderPlaced {
  reserved 3;                 // amount was removed; its number is retired
  reserved "amount";          // the old name is retired as well

  string order_id    = 1;
  string customer_id = 2;
  string currency    = 4;
  int64  amount_cents = 5;    // new field, new number, never reuse 3
}

Upgrade order in practice

The compatibility mode dictates not only which changes are legal but also the sequence in which the two sides of a stream must be deployed. Under BACKWARD, the new schema is designed so that new consumers can read data still being produced under the old schema; consumers are therefore upgraded first, after which producers can move to the new schema safely. Under FORWARD, the arrangement reverses: producers move first because the guarantee is that old consumers can still read the newly produced data. Under FULL, both guarantees hold, so the two sides can be rolled out in any order. Kafka Streams applications, which both consume and produce, are supported only under BACKWARD and the modes that include it, namely FULL and FULL_TRANSITIVE (Confluent schema-evolution documentation, as of 2026-07-17).

Operational Practices for Schema Evolution

Choosing and operating a compatibility policy is as much an organizational decision as a technical one. The following practices reflect the way schema registries are commonly run in production.

Choosing a compatibility mode

The right mode depends on which side of the stream tends to deploy first and how tightly the two sides are coordinated. When consumers are numerous and slow to upgrade, as with a widely subscribed event, BACKWARD or BACKWARD_TRANSITIVE lets producers evolve without waiting for every consumer. When a single producer feeds many independent consumers that cannot be upgraded in lockstep, FORWARD protects those consumers from a producer change. FULL and FULL_TRANSITIVE impose the strictest discipline and are appropriate for long-lived, high-value topics where either side may change at any time. The transitive variants trade a small amount of flexibility for a strong guarantee: because they validate against the entire version history, they prevent a slow drift that a sequence of latest-only checks would permit.

Testing compatibility in continuous integration

Because the registry enforces compatibility at registration time, a broken schema change is most cheaply caught before deployment. The registry exposes a compatibility-check endpoint that reports whether a candidate schema would be accepted under the subject’s configured mode, without registering it. Wiring this call into a continuous-integration pipeline turns a potential production incident into a failed build:

# Check a candidate schema against the latest version of a subject.
# Returns {"is_compatible": true} or false without registering.
curl -s -X POST \
  http://schema-registry:8081/compatibility/subjects/orders-value/versions/latest \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d @candidate_schema_request.json

Treating schema definitions as versioned source code, reviewed and tested like any other artifact, is the connective tissue between the registry and a broader data-contract discipline. The same schema that governs the wire format can drive downstream typing in transformation frameworks, so that a well-typed event flows into modeled tables; the transformation layer is discussed in the overview of building transformation pipelines with dbt.

Handling genuinely breaking changes

Some changes cannot be made compatibly. Changing a field’s type in an incompatible way, or removing a required field that consumers depend on, will and should be rejected under any checked mode. Two disciplined options exist. The first is to publish the incompatible data to a new topic and migrate consumers deliberately, retiring the old topic once no consumer remains. The second, to be used sparingly, is to set the subject to NONE for a controlled window, coordinate the producer and consumer deployment manually, and restore a checked mode afterward. Setting NONE permanently removes the very protection the registry exists to provide and is not a substitute for a migration plan.

Platform context

The registry evolves alongside the broader Kafka platform. Confluent Platform 8.0 reached general availability in June 2025 and bundles Apache Kafka 4.0, which removes the dependency on ZooKeeper in favor of the KRaft consensus protocol; the same release brought general availability of client-side field-level encryption and of passwordless authentication for the Schema Registry (Confluent Platform 8.0 announcement, as of 2026-07-17). These platform-level changes do not alter the wire format or the compatibility semantics described here, which remain stable by design, but they do affect how a registry is secured and operated.

Frequently Asked Questions

Is the default compatibility mode BACKWARD or FULL?

The default compatibility mode in the Confluent Schema Registry is BACKWARD. Under BACKWARD, a candidate schema is validated only against the latest registered version, and it permits deleting fields and adding fields that carry a default value. It does not provide the two-directional guarantee of FULL, nor the whole-history guarantee of the transitive variants. Teams that need protection against drift across the entire version history should explicitly configure a transitive mode.

When are compatibility checks actually performed?

Compatibility is enforced at schema-registration time, not when an individual message is decoded. When a producer or a deployment pipeline attempts to register a new schema under a subject, the registry compares it against the versions required by the configured mode and rejects the registration if it would break compatibility. A rejected registration prevents the incompatible schema, and therefore any message written with it, from entering the log.

How does the wire format differ between Avro and Protobuf?

Both use a 5-byte header consisting of a magic byte set to 0x00 followed by a 4-byte big-endian schema ID. Protobuf inserts an additional segment, a length-prefixed message-index array of variable-length integers, between the schema ID and the payload, because a single .proto file may define several message types and the deserializer must know which one was serialized. Avro and JSON Schema do not include this segment.

Should producers or consumers be upgraded first?

The order follows the compatibility mode. Under BACKWARD, consumers are upgraded first, because the guarantee is that new consumers can read data still produced under the old schema. Under FORWARD, producers are upgraded first, because old consumers must be able to read newly produced data. Under FULL, both guarantees hold and the two sides may be upgraded in either order.

Why must a Protobuf field number never be reused?

In the Protobuf binary encoding, each value is identified by its field number rather than its name. A decoder holding data written with an older schema will interpret bytes according to whatever field the number denoted at the time. If the number is later reassigned to a different field, especially with a different type, the old data is misread and silently corrupted. Marking a removed number as reserved permanently blocks its reuse and prevents this class of error.

References

• Apache Avro Specification — Schema Resolution
• Confluent — Schema Evolution and Compatibility
• Confluent — Formats, Serializers, and Deserializers (wire format, subject naming)
• Protocol Buffers — Best Practices (field numbers, reserved)
• Apache Avro 1.12.1 Release Announcement

Conclusion

The schema registry solves a problem that is easy to overlook until it causes an outage: a Kafka message carries data but not the description needed to interpret it. By storing each schema once, assigning it a compact identifier, and enforcing compatibility rules at registration time, the registry turns an implicit and fragile agreement between independently deployed producers and consumers into an explicit, versioned contract. The 5-byte wire format is the small piece of framing that makes this work on every message, with Protobuf’s message-index array the one notable variation to keep straight.

The two dominant binary formats reach compatibility by different means that are worth internalizing separately. Avro reconciles a reader schema against a writer schema, with field defaults and aliases as the levers that make additive and rename changes safe. Protobuf anchors compatibility on stable field numbers, with the reserved declaration as the guard against the corrupting error of number reuse. Above both sits the compatibility-mode matrix, whose central practical consequence is the upgrade order: BACKWARD upgrades consumers first, FORWARD upgrades producers first, and FULL frees the two sides to move independently. Applied with a default of BACKWARD understood correctly, compatibility checks wired into continuous integration, and a deliberate plan for the changes that cannot be made compatibly, the registry becomes the mechanism that lets a streaming data model evolve for years without breaking the systems that depend on it.

Apache Spark presents a deceptively simple programming surface. A data engineer writes a few DataFrame transformations or a SQL query, calls an action, and a result appears. Underneath that surface sits a distributed query engine that parses the request into a logical plan, rewrites it through dozens of optimization rules, compiles fragments of it to Java bytecode, and coordinates hundreds of tasks across a cluster. When a job runs slowly, the cause is almost never the API call that was written. It lies in how the engine translated that call into stages, how much data crossed the network during a shuffle, and whether the optimizer chose a physical plan suited to the actual shape of the data.

This article traces that translation end to end. It assumes familiarity with running Spark jobs and concentrates on the execution internals that determine performance: the DAG scheduler and its stage boundaries, the Catalyst optimizer and its adaptive re-planning, the Tungsten engine and its code generation, the sort-based shuffle, and the join strategies that Spark selects. The version referenced throughout is the Spark 4.1.x line, current as of mid-2026, with Spark 4.1.2 released on 21 May 2026 and the 4.0 maintenance line still active through Spark 4.0.3 (Apache Spark news and downloads, as of 2026-07-14).

The Spark Execution Model: Jobs, Stages, and Tasks

A Spark application is coordinated by a single process called the driver, which holds the program logic, and a set of executor processes distributed across the cluster, which perform the actual computation on partitions of data. A partition is the unit of parallelism — a contiguous slice of a dataset that a single task processes. The driver builds the plan; the executors run it.

Spark evaluates transformations lazily. Operations such as map, filter, select, and join do not trigger computation; they extend a logical description of the work. Computation begins only when an action — for example count, collect, or write — is invoked. At that moment the driver submits a job, and the DAG scheduler translates the accumulated transformations into an execution plan structured as a directed acyclic graph (DAG) of stages.

Narrow and Wide Dependencies

The DAG scheduler divides a job into stages by examining the dependencies between partitions. A narrow dependency is one in which each parent partition contributes to at most one child partition. Transformations such as map and filter are narrow: a task can process its input partition and produce its output partition without consulting any other partition. Narrow dependencies are pipelined — chained together and executed within a single stage without materializing intermediate results.

A wide dependency is one in which a child partition depends on multiple parent partitions. Transformations such as groupByKey, reduceByKey, join, distinct, and repartition are wide, because a given output key may be assembled from records scattered across every input partition. A wide dependency cannot be pipelined; it requires data to be redistributed across the cluster so that all records sharing a key land in the same partition. That redistribution is the shuffle, and every shuffle defines a stage boundary (Apache Spark documentation; SparkInternals, as of 2026-07-14).

Figure 1: A job is split into stages at each shuffle. Narrow operators are pipelined inside a stage; each stage runs one task per partition.

Within a stage, the unit of execution is the task. Spark launches one task per partition, and tasks within a stage run in parallel across the available executor cores. The number of tasks in a stage therefore equals the number of partitions, which is why partition count directly governs both parallelism and per-task workload. A stage with too few partitions underuses the cluster; a stage with too many partitions incurs scheduling overhead for tasks that each process a trivial amount of data.

The Catalyst Optimizer and Adaptive Query Execution

Before any task runs, the query passes through Catalyst, Spark SQL’s query optimizer. Catalyst is an extensible optimizer built on functional tree-transformation constructs in Scala. It represents a query as a tree of nodes and rewrites that tree by applying rules — pattern-matching functions that transform one tree into an equivalent, cheaper tree. The DataFrame and SQL APIs both compile down to the same Catalyst representation, so an SQL query and its DataFrame equivalent are optimized identically.

The Four Phases

Catalyst structures optimization in four phases (Databricks, “Deep Dive into Spark SQL’s Catalyst Optimizer,” 2015-04-13):

1. Analysis. The parser produces an unresolved logical plan in which column and table names are still symbolic. The analyzer resolves these references against the catalog — the registry of tables, columns, and their types — turning the tree into an analyzed logical plan with fully typed attributes.
2. Logical optimization. A set of rule-based rewrites is applied to reduce work. Named optimizations include predicate pushdown (moving filters as close to the data source as possible), projection pruning (reading only the columns a query needs), constant folding (evaluating constant expressions once at plan time), and join reordering. The output is an optimized logical plan.
3. Physical planning. Catalyst generates one or more physical plans that specify concrete operators — which join algorithm to use, how to exchange data — and selects among them by cost. This is the only cost-based phase; all others are purely rule-based.
4. Code generation. The selected physical plan is compiled, in part, to JVM bytecode for execution, a step handled by the Tungsten engine and discussed in the next section.

The design of Catalyst was first described in the academic literature by Armbrust and colleagues in “Spark SQL: Relational Data Processing in Spark,” presented at SIGMOD 2015. That paper introduced the tree-transformation framework that remains the foundation of Spark SQL a decade later.

Figure 2: Catalyst’s four phases. Predicate pushdown into Parquet scans and projection pruning are rule-based; join-algorithm selection is the cost-based decision.

Predicate pushdown and projection pruning are the phases where storage format matters most. When Spark reads a columnar file, it can skip entire column chunks and row groups that a query does not touch, so the physical layout of the source determines how much of Catalyst’s logical optimization can be realized as reduced I/O. The mechanics of that pushdown are covered in the companion discussion of Apache Parquet and Apache Arrow internals, which explains how row-group statistics and column pruning let the scan read only the bytes a query requires.

Adaptive Query Execution

Catalyst’s cost-based decisions historically relied on statistics gathered before execution — table sizes, column histograms — which are often stale, missing, or wrong for intermediate results. A join planned as a sort-merge because both sides looked large may, after a filter, involve one tiny side. Adaptive Query Execution (AQE) addresses this by re-optimizing the plan at runtime using statistics measured from completed shuffles. AQE has been enabled by default since Spark 3.2.0 through the configuration flag spark.sql.adaptive.enabled (change tracked in SPARK-33679); it existed but was off by default in Spark 3.0 and 3.1 (Spark SQL Performance Tuning documentation, as of 2026-07-14).

AQE has three headline capabilities. It dynamically coalesces shuffle partitions, merging contiguous small partitions after a shuffle so that a fixed partition count no longer produces hundreds of tiny tasks. It switches join strategies, upgrading a planned sort-merge join to a broadcast hash join once a shuffle reveals that one side is small enough. And it handles skew by splitting oversized partitions in a sort-merge join at runtime. Because AQE inserts these decisions between stages — precisely where actual data sizes become known — it corrects the very estimation errors that make static planning brittle.

Figure 3: AQE re-optimizes at each shuffle boundary — coalescing partitions, switching join strategies, and splitting skewed partitions using measured statistics.

Tungsten and Whole-Stage Code Generation

Where Catalyst decides what to execute, Project Tungsten governs how efficiently each task runs on a single core. Tungsten began in Spark 1.6 as an initiative to bring Spark’s execution closer to bare-metal CPU and memory efficiency. Its stated goals were four: explicit off-heap memory management, cache-aware computation, code generation, and reduced virtual function calls (Databricks, “Project Tungsten,” 2015-04-28). The second generation of Tungsten, including whole-stage code generation, shipped in Spark 2.0 (Databricks, “Apache Spark as a Compiler,” 2016-05-23).

The UnsafeRow Binary Format

A conventional JVM program represents each row as a tree of Java objects, each carrying object headers, pointers, and padding, all managed by the garbage collector. For a dataset of billions of rows, that overhead dominates both memory consumption and CPU time spent in garbage collection. Tungsten replaces this with UnsafeRow, a compact binary layout in which a row is stored as a contiguous block of bytes. Because a Dataset has a known schema, Tungsten can lay each field out at a fixed offset and manage the backing memory off-heap — outside the region the garbage collector scans — sidestepping JVM object overhead and garbage-collection pressure entirely (The Internals of Spark SQL; Databricks, “Project Tungsten”).

Figure 4: UnsafeRow stores a row as a contiguous off-heap byte block with a null bitset, a fixed-length region, and a variable-length region, avoiding JVM object overhead.

Whole-Stage Code Generation

The classic way to execute a query plan is the Volcano iterator model, in which each operator implements a next() method and pulls rows one at a time from its child. This model is general but slow: every row crosses a virtual function call at every operator, intermediate values are boxed and materialized, and the CPU cannot keep data in registers. Whole-stage code generation (WSCG), tracked as SPARK-12795, collapses an entire chain of operators within a stage into a single JVM function. Instead of many operators each calling the next, Spark generates one tight loop that applies the whole chain of narrow operations to each row, eliminating virtual dispatch and keeping intermediate values in CPU registers rather than materializing rows. The generated Java source is compiled to bytecode at runtime by the Janino compiler (Databricks, “Apache Spark as a Compiler,” 2016-05-23; The Internals of Spark SQL).

Alongside WSCG, Spark 2.0 introduced vectorized columnar reads for Parquet (SPARK-12992), in which the reader decodes a batch of column values at a time rather than row by row, matching the columnar layout of the file to the batch-oriented execution engine. The scan and the compute path both operate on batches, which keeps the CPU’s instruction and data caches warm.

Figure 5: WSCG replaces a chain of Volcano-model operators, each with a virtual next() call, with a single generated loop that keeps intermediate values in registers.

Databricks demonstrated the effect of these techniques with a benchmark headlined as joining “one billion rows per second on a laptop.” That figure is a specific micro-benchmark result rather than a general guarantee, and it should be read as an illustration of what fused execution can achieve on a favorable workload, not a number any arbitrary job will reach.

The Shuffle and Data Skew

The shuffle is the operation that redistributes data across the cluster so that all records sharing a key reside on the same partition. It is triggered by every wide dependency and is, in most jobs, the single largest consumer of time and resources, because it combines serialization, disk writes, network transfer, and disk reads. Spark has used a sort-based shuffle as the default since Spark 1.2, when spark.shuffle.manager=sort replaced the earlier hash-based shuffle. The SortShuffleManager is now the only shuffle manager in vanilla Spark; it dispatches to three internal write paths — BypassMergeSortShuffleHandle, SerializedShuffleHandle, and BaseShuffleHandle — depending on the number of partitions and whether map-side aggregation is needed (apache/spark source, SortShuffleManager.scala, as of 2026-07-14).

Map Side and Reduce Side

A shuffle has two halves. On the map side, each task in the upstream stage processes its input partition and assigns every output record to a target partition, determined by hashing the key. Records accumulate in an in-memory structure — a PartitionedAppendOnlyMap when aggregation is required — grouped by target partition. When the structure exhausts its memory budget, the task spills a sorted run to disk, sorting with TimSort. At the end of the task, the spilled runs and any remaining in-memory records are merged into a single shuffle file with an accompanying index that marks where each partition’s bytes begin.

On the reduce side, each task in the downstream stage fetches the byte ranges destined for its partition from every map output across the cluster, then merges those streams on the fly using a min-heap. This on-the-fly merge distinguishes Spark’s shuffle from Hadoop MapReduce, which materializes a fully merged file on disk before the reduce begins. The volume of data fetched during this phase, multiplied by network latency, is what makes the shuffle expensive.

Figure 6: The sort-based shuffle. Map tasks partition and spill sorted runs; reduce tasks fetch their partition from every map output and merge on the fly.

The number of reduce-side partitions is governed by spark.sql.shuffle.partitions, whose default is 200. This fixed default is frequently wrong for a given data size: for a small result it creates 200 tiny tasks with more scheduling overhead than work, and for a large result it creates 200 oversized tasks that spill repeatedly to disk (Spark 4.1.2 SQL Performance Tuning documentation, as of 2026-07-14). AQE’s partition coalescing exists precisely to correct the small-partition case automatically, merging contiguous partitions toward a target size rather than requiring the value to be tuned by hand for every job.

Data Skew

Data skew occurs when records are distributed unevenly across keys, so that a few partitions hold far more data than the rest. Because a stage completes only when its slowest task finishes, a single oversized partition can hold up an entire job while the cluster sits idle. Skew is the most common reason a stage that processes a moderate dataset runs for an unexpectedly long time. It typically arises from a highly frequent key — a null join key, a default value, or a dominant category.

Two mitigations are standard. The first is salting: a random suffix is appended to the skewed key so that its records spread across many partitions, the join or aggregation is performed on the salted key, and the results are combined afterward. Salting is explicit and always available, but it requires rewriting the query. The second is AQE skew-join handling, enabled by default through spark.sql.adaptive.skewJoin.enabled, which detects an oversized partition at runtime and splits it into sub-partitions automatically, replicating the matching side of the join. A partition is treated as skewed when it exceeds spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes — default 256 MB — and also exceeds a configurable factor times the median partition size.

Figure 7: A single hot key stalls a stage. Salting spreads the key manually; AQE splits the skewed partition automatically at the 256 MB threshold.

Join Strategies and Practical Performance Tuning

The join is where the cost of Catalyst’s physical planning, the shuffle, and AQE all converge. Spark chooses among three principal join algorithms, and the choice is governed largely by data size.

Broadcast, Sort-Merge, and Shuffle-Hash

A broadcast hash join is chosen when one side of the join is small — below spark.sql.autoBroadcastJoinThreshold, whose default is 10485760 bytes (10 MB). The small side is collected to the driver and broadcast to every executor, which builds a hash table from it in memory. The large side is then joined locally, partition by partition, with no shuffle of the large side at all. This is the cheapest join when it applies, because it avoids redistributing the large table.

A sort-merge join is the general default for two large inputs. Both sides are shuffle-partitioned on the join key and sorted, and the sorted partitions are then merged. It requires a full shuffle of both sides, which makes it expensive, but it scales to inputs far larger than memory because it never builds a full hash table.

A shuffle-hash join shuffles both sides on the join key but then builds an in-memory hash table on one side rather than sorting. AQE can convert a sort-merge join into a shuffle-hash join when all post-shuffle partitions are small enough to build hash tables locally, governed by spark.sql.adaptive.maxShuffledHashJoinLocalMapThreshold. AQE can also upgrade a planned sort-merge join to a broadcast hash join once the actual shuffle-output size of one side proves to be below the broadcast threshold (Spark 4.1.2 SQL Performance Tuning documentation, as of 2026-07-14).

Figure 8: Join-strategy selection. A side under the 10 MB threshold triggers a broadcast; otherwise Spark shuffles both sides and chooses shuffle-hash or sort-merge, with AQE able to switch at runtime.

The following table summarizes the trade-offs. A broadcast join is preferred whenever a side genuinely fits the threshold, because it eliminates the most expensive part of the join.

Configuration Defaults That Govern Performance

A small set of configuration values controls most of the behavior described above. The defaults below are those documented for the Spark 4.1.2 line (Spark SQL Performance Tuning documentation, as of 2026-07-14). Understanding them is more useful than memorizing them, because AQE now adjusts several of these dynamically.

Practical Tuning Priorities

Several practical conclusions follow directly from the mechanics above. The first is that reducing the amount of data read is the cheapest optimization available, because bytes never read are bytes never shuffled. Storing data in a columnar format with row-group statistics lets Catalyst’s predicate pushdown and projection pruning skip most of the file before any compute begins; the same principle underlies the design of modern open table formats such as Iceberg, Delta Lake, and Hudi, which add partition pruning and file-level statistics on top of the columnar layer.

The second is that the shuffle should be minimized, not merely tuned. Filtering before a join, pre-aggregating where possible, and enabling broadcast joins for small dimension tables all reduce the volume of shuffled data. Where a shuffle is unavoidable, AQE’s partition coalescing removes most of the need to hand-tune spark.sql.shuffle.partitions, though a very large job may still benefit from a higher explicit value so that individual tasks do not spill.

The third is that skew deserves direct attention. Because a stage is bounded by its slowest task, a job that appears to under-use the cluster is often waiting on one hot partition. The Spark UI’s stage view, which shows the distribution of task durations and shuffle read sizes, is the fastest way to confirm skew before applying salting or verifying that AQE skew handling engaged.

Spark occupies a specific position in the broader landscape of data systems. It is a distributed, disk-spilling engine designed for datasets that exceed the memory of any single machine, which is a different design point from the single-node vectorized engines examined in the comparison of lakehouse query engines such as Trino, StarRocks, and DuckDB. It is also predominantly a batch and micro-batch engine rather than a low-latency streaming system, a distinction developed in the discussion of streaming versus batch processing architectures. In practice Spark is frequently the compute layer beneath a SQL-first transformation workflow, and the models produced by tools such as dbt for transformation pipelines often compile down to the very Catalyst plans and shuffles described here.

Frequently Asked Questions

What is the difference between Catalyst and Tungsten in Spark?

Catalyst and Tungsten operate at different levels. Catalyst is the query optimizer: it decides what to execute by transforming a logical plan through analysis, rule-based logical optimization, and cost-based physical planning. Tungsten is the execution engine: it decides how efficiently each task runs on a CPU by generating fused machine-friendly code, storing rows in a compact off-heap binary format called UnsafeRow, and managing memory outside the garbage collector. Catalyst produces the plan; Tungsten executes it fast.

Why is the shuffle so expensive in Spark?

A shuffle redistributes data across the cluster so that all records sharing a key land on the same partition. It combines four costly operations: serializing records, spilling sorted runs to disk when memory is exhausted, transferring bytes across the network, and merging the fetched streams. Because it is triggered by every wide dependency and moves data proportional to dataset size, the shuffle is usually the dominant cost of a Spark job, which is why reducing the amount of shuffled data is the highest-value optimization.

Is Adaptive Query Execution enabled by default?

Yes. Adaptive Query Execution has been enabled by default since Spark 3.2.0 through the configuration flag spark.sql.adaptive.enabled. It was available but off by default in Spark 3.0 and 3.1. AQE re-optimizes a plan at runtime using statistics from completed shuffles, coalescing small partitions, switching a sort-merge join to a broadcast join when a side turns out small, and splitting skewed partitions automatically.

What does spark.sql.shuffle.partitions control, and why is 200 often wrong?

It sets the number of post-shuffle partitions for SQL and DataFrame joins and aggregations, and its default is 200. The value is fixed regardless of data size, so it is frequently a poor fit: for a small result it creates 200 tiny tasks whose scheduling overhead exceeds their work, and for a large result it creates 200 oversized tasks that spill to disk. AQE’s partition coalescing corrects the small-partition case automatically by merging contiguous partitions toward a target size, which reduces the need to tune the value by hand.

When does Spark use a broadcast join instead of a sort-merge join?

Spark chooses a broadcast hash join when one side of the join is smaller than spark.sql.autoBroadcastJoinThreshold, whose default is 10 MB. The small side is broadcast to every executor and the large side is joined locally without being shuffled, which is the cheapest join when it applies. Two large inputs default to a sort-merge join, which shuffles and sorts both sides. With AQE, a planned sort-merge join can be upgraded to a broadcast join at runtime once the actual size of one side is known to fall below the threshold.

Conclusion

Spark’s performance is best understood as the product of three cooperating layers. Catalyst decides the shape of the computation, rewriting a query through rule-based logical optimization and one cost-based physical-planning phase, and since Spark 3.2.0 it re-optimizes that plan at runtime through Adaptive Query Execution. Tungsten then executes each task efficiently, fusing operators into generated code that shipped in Spark 2.0 and storing rows in a compact off-heap format that avoids garbage-collection overhead. Between the stages sits the shuffle, the sort-based data-redistribution mechanism that has been the default since Spark 1.2 and that dominates the cost of most jobs.

For a practicing data engineer, the practical consequences are consistent. The largest gains come from reading less data, shuffling less data, and eliminating skew — in that order — because each addresses a cost that the engine cannot optimize away on its own. The configuration defaults matter less than the model behind them: once the boundary between narrow and wide dependencies, the role of the broadcast threshold, and the behavior of AQE are clear, the Spark UI and the physical plan become sufficient to diagnose almost any slow job. Reasoning about how the engine executes a query, rather than adjusting parameters in isolation, is what turns Spark performance work from guesswork into engineering.

References

• Apache Spark, “Spark SQL Performance Tuning” (official documentation, latest / 4.1.x). spark.apache.org/docs/latest/sql-performance-tuning.html
• Apache Spark, “News and Releases” (version verification, 4.1.2 / 4.0.3, mid-2026). spark.apache.org/news
• Databricks, “Deep Dive into Spark SQL’s Catalyst Optimizer” (2015-04-13). databricks.com/blog
• Databricks, “Project Tungsten: Bringing Apache Spark Closer to Bare Metal” (2015-04-28). databricks.com/blog
• Databricks, “Apache Spark as a Compiler: Joining a Billion Rows per Second on a Laptop” (2016-05-23). databricks.com/blog
• Armbrust et al., “Spark SQL: Relational Data Processing in Spark,” SIGMOD 2015 — the paper that introduced Catalyst.

A modern data lakehouse separates the data it stores from the software that computes over it. Table formats such as Apache Iceberg or Delta Lake define how records, partitions, and transactions are laid out as files on object storage, but they do not execute queries. That work belongs to a distinct tier: the query engine. Because storage and compute are decoupled in a lakehouse, an organization can place several different engines over the same files and choose among them per workload. This guide examines three query engines that occupy very different points in that design space — Trino, StarRocks, and DuckDB — and explains how their architectures, deployment models, pushdown behavior, and caching strategies determine which one fits a given use case. The comparison is deliberately confined to the compute tier; the underlying file and table formats are treated as a fixed substrate rather than re-derived here.

The engines were chosen because they represent three archetypes rather than three interchangeable products. Trino is a distributed, federated massively parallel processing (MPP) engine built for ad-hoc SQL across many sources. StarRocks is an MPP online analytical processing (OLAP) engine designed for interactive, high-concurrency serving on the lakehouse. DuckDB is a single-node, in-process engine that queries lakehouse tables directly from inside a host program without any cluster. Understanding why each exists, and where each stops being the right tool, is more durable than memorizing a feature list, because the storage-compute separation that makes the lakehouse attractive also makes engine selection a recurring decision rather than a one-time commitment.

The Query-Engine Tier of the Lakehouse

A lakehouse is a data architecture that places a transactional table layer over inexpensive object storage so that a single copy of the data serves both exploratory analytics and production reporting. Its defining property is the separation of storage from compute. Data files, most often Apache Parquet, sit in an object store such as Amazon S3. A table format layered on top of those files records which files belong to a table, how they are partitioned, and which version of the table is current. Query engines then read that table format and execute SQL. Because no engine owns the data, several can read the same tables concurrently, and an organization can add or replace an engine without migrating a single byte.

This decoupling is what distinguishes a lakehouse from a classical data warehouse, in which storage and compute are fused inside one proprietary system. The trade-off is that reading data now crosses a network boundary to object storage, which has higher and more variable latency than a local disk. Much of what differentiates the engines in this comparison is how they cope with that boundary: how aggressively they avoid reading files at all, and how they cache the files they must read. The table format itself is out of scope here; readers who need the storage-substrate details can consult the companion analysis of Iceberg, Delta Lake, and Hudi table formats, and the treatment of the file layer in the guide to Parquet and Apache Arrow columnar storage.

Two mechanisms recur throughout the comparison and are worth defining once. Predicate pushdown is the practice of pushing a query’s filter conditions down to the storage layer so that files, partitions, or row groups that cannot match are never read. Projection pushdown reads only the columns a query references, which is efficient because Parquet stores data column by column. Both reduce the volume of bytes crossing the network boundary, and the degree to which an engine exploits them is a primary determinant of its latency over object storage.

Trino: Federated Distributed MPP

Trino is a distributed SQL query engine designed for interactive and ad-hoc analytics across large and heterogeneous datasets. It follows the massively parallel processing model, meaning a query is divided into parallel units of work that execute simultaneously across many machines. Trino uses date-independent sequential release numbers and ships frequently; the latest release at the time of writing is Trino 482, published on 2026-06-25 (github.com/trinodb/trino/releases; trino.io, as of 2026-07-11). Its defining characteristic is federation: a single Trino cluster can query many different data sources in one statement, joining a lakehouse table against a relational database without moving either.

Coordinator and Workers

A Trino cluster consists of one coordinator and zero or more workers. The coordinator is the brain: it receives SQL statements from clients, parses and analyzes them, builds a distributed execution plan, and schedules and monitors the work assigned to the workers. Workers execute the tasks they are given and process the underlying data. Each node runs as a single Java Virtual Machine (JVM) process and achieves parallelism through many threads within that process. A cluster with more workers can process more data concurrently, which is how Trino scales: an operator adds worker nodes to raise throughput rather than replacing the cluster with a larger one (trino.io/docs/current/overview/concepts.html, as of 2026-07-11).

The internal decomposition of a query is precise and worth following, because it explains both Trino’s parallelism and its memory behavior. A client submits a statement, the text of the SQL. Trino turns it into a query, the running instance of that statement. The query plan is a tree of stages, each of which is realized as a set of tasks distributed across workers. Each task processes splits, which are addressable sections of the input data, using a pipeline of drivers and operators that apply the actual relational logic. Data moves between stages through exchanges. This structure is the classic distributed, federated MPP execution model, and it allows a single large scan to be spread across every worker in the cluster.

Connectors and Catalogs

Trino’s federation rests on two concepts. A connector adapts Trino to a particular kind of data source; connectors exist for Hive, Delta Lake, Iceberg, PostgreSQL, and many other systems. A catalog is a configured instance of a connector pointing at a specific data source. Multiple catalogs coexist in one cluster, and because a query can reference tables from more than one catalog, Trino can join across sources in a single statement. The namespace is three levels deep: catalog, then schema, then table (trino.io/docs/current/overview/concepts.html, as of 2026-07-11). A catalog is defined by a small properties file. The following configures an Iceberg catalog backed by a REST catalog service and S3 storage.

# etc/catalog/lakehouse.properties
connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=https://catalog.example.com/api/catalog
fs.native-s3.enabled=true
s3.region=us-east-1
# turn on the local file-system cache for this catalog
fs.cache.enabled=true
fs.cache.directories=/mnt/trino-cache

With this catalog registered, a query such as SELECT * FROM lakehouse.sales.orders o JOIN postgres.crm.customers c ON o.cust_id = c.id reads the orders table from Iceberg on S3 and joins it against a live PostgreSQL table, with the join executed in the Trino cluster. This is the federation capability that makes Trino a shared query gateway over a mixture of lakehouse tables and operational databases. It also positions Trino as a convenient execution engine beneath a transformation framework; a project that models tables with the dbt transformation tool can compile its SQL against a Trino target and reach every registered catalog through one connection.

Pushdown and Caching in Trino

For lakehouse tables, Trino’s Iceberg connector applies predicate pushdown to drive partition and file pruning, so that a filtered query skips partitions and data files that cannot satisfy the condition, and projection pushdown to read only the referenced columns. Vectorized readers handle both Parquet and ORC file formats (Trino Iceberg connector documentation, trino.io, as of 2026-07-11). Pushdown reduces work but does not remove the cost of repeatedly fetching the same files from object storage. To address that, Trino provides a file-system cache for the Delta Lake, Hive, and Iceberg connectors, with Hudi support noted as forthcoming, and native support for Alluxio as a distributed caching layer in front of object storage, enabled with fs.alluxio.enabled=true. The older Rubix caching path has been superseded by the native file-system cache (trino.io/docs/current/object-storage/file-system-cache.html and file-system-alluxio.html, as of 2026-07-11).

StarRocks: MPP OLAP Serving on the Lakehouse

StarRocks is an MPP OLAP engine oriented toward interactive analytics: sub-second dashboards, high-concurrency BI, and repeated queries over lakehouse tables. It is a Linux Foundation project (github.com/StarRocks/starrocks, as of 2026-07-11). The latest major release is StarRocks 4.0, released on 2025-10-17. The vendor reports roughly 60 percent faster query execution year over year on the TPC-DS benchmark for 4.0; this is a self-reported figure and should be read alongside the benchmark caveat below (starrocks.io/blog/starrocks-2025-year-in-review, as of 2026-07-11). Where Trino optimizes for breadth of sources, StarRocks optimizes for the speed and concurrency of repeated queries over a smaller set of well-modeled tables.

Frontend, Backend, and Compute Nodes

StarRocks also follows the MPP model but uses two primary node types. The Frontend (FE) manages metadata, handles client connections, parses SQL, and performs query planning and scheduling. The Backend (BE) stores data and executes the query fragments assigned to it against its local storage. A third node type, the Compute Node (CN), is effectively a Backend without attached storage, used to separate compute from storage (docs.starrocks.io/docs/introduction/Architecture/, as of 2026-07-11). The division of labor resembles Trino’s coordinator-and-worker split, but StarRocks integrates a storage role into its Backend nodes, which is central to its two deployment modes.

In shared-nothing mode, Backends hold table data on their own local storage, which yields the lowest read latency but couples storage capacity to compute capacity. In shared-data mode, the Frontend coordinates a fleet of stateless Compute Nodes while the authoritative data lives on object storage or HDFS, with a local cache smoothing access. Shared-data mode is StarRocks’ answer to storage-compute separation: it retains the elasticity of a lakehouse, in which compute can scale independently, while keeping a warm local cache so that repeated queries do not pay full object-storage latency each time (docs.starrocks.io/docs/introduction/Architecture/, as of 2026-07-11).

Optimizer, Materialized Views, and Data Cache

StarRocks pairs a fully vectorized execution engine and columnar storage with a cost-based optimizer (CBO), which estimates the cost of alternative execution plans and chooses the cheapest rather than following the query’s written order (starrocks.io/blog/introduction_to_starrocks, as of 2026-07-11). On top of this sits the feature most responsible for its interactive performance on the lakehouse: materialized views (MVs). A materialized view is a precomputed, stored result of a query that the engine can substitute for the original computation. StarRocks supports multi-column partitioned MVs that align with Iceberg or Hive partitions, which allows incremental refresh — only the changed partitions are recomputed rather than the whole view. The cost-based optimizer performs transparent query rewriting, redirecting an incoming query to an eligible MV without the author referencing it. The vendor reports large MV-driven speedups on eligible queries — by an order of magnitude or more; this is a self-reported figure (starrocks.io/blog/starrocks-2025-year-in-review; docs.starrocks.io async_mv documentation, as of 2026-07-11).

An external catalog connects StarRocks to a lakehouse without ingesting the data, and a materialized view built on that catalog becomes the acceleration layer. The following sketch registers an Iceberg catalog and defines an MV that the optimizer can transparently rewrite queries onto.

-- Register the lakehouse as an external catalog
CREATE EXTERNAL CATALOG iceberg_cat
PROPERTIES (
  "type" = "iceberg",
  "iceberg.catalog.type" = "rest",
  "iceberg.catalog.uri" = "https://catalog.example.com/api/catalog"
);

-- Precompute a daily rollup, partition-aligned for incremental refresh
CREATE MATERIALIZED VIEW sales_daily
PARTITION BY order_date
REFRESH ASYNC
AS
SELECT order_date, region, sum(amount) AS revenue
FROM iceberg_cat.sales.orders
GROUP BY order_date, region;
-- Queries that match this pattern are rewritten onto the MV by the CBO

Beneath the MV layer, StarRocks provides a Data Cache that caches blocks of remote files locally to smooth the latency and jitter of object storage. StarRocks 4.0 added metadata caching, compaction, and file bundling that the vendor reports reduce cloud API calls by up to 90 percent, and strengthened Iceberg support with hidden-partition handling, faster metadata parsing, a new compaction API, and native Iceberg table writes; 4.0 also introduced JSON as a first-class type, reported at 3 to 15 times faster on JSON queries. These are self-reported figures (starrocks.io/blog/starrocks-2025-year-in-review, as of 2026-07-11).

DuckDB and DuckLake: In-Process Lakehouse Analytics

DuckDB occupies the opposite end of the spectrum from Trino and StarRocks. It is a single-node, in-process, vectorized analytical engine — often described as SQLite for analytics — that runs inside the host process rather than as a separate cluster. There is no coordinator, no worker fleet, and no network protocol between the application and the engine; a query executes in the same process as the program that issued it. The current stable line is DuckDB 1.5.x, with v1.5.3 shipping on 2026-05-20 (duckdb.org, as of 2026-07-11). This design removes distributed-systems overhead entirely, at the cost of being bounded by one machine’s memory and cores, with no distributed shuffle. DuckDB’s role as a general in-process SQL and dataframe engine is examined in the companion comparison of DuckDB and Polars for in-process analytics; the concern here is narrower — its role as a lakehouse engine.

Lakehouse Formats as First-Class Citizens

DuckDB reads and writes lakehouse tables through native extensions rather than third-party libraries. The iceberg extension handles Apache Iceberg, the delta extension handles Delta Lake, the ducklake extension handles DuckLake, and Lance is also supported. Implementing these as native extensions rather than external bindings lets DuckDB apply complex filter pushdowns and manage memory carefully during scans (duckdb.org/docs/current/lakehouse_formats, as of 2026-07-11). The Iceberg extension can also connect to Iceberg REST catalogs, such as AWS Glue, Unity-style, or Lakekeeper services, so that DuckDB resolves tables through the same catalog a cluster engine would use (duckdb.org iceberg REST catalog documentation, as of 2026-07-11). Version 1.5.3 extended the Iceberg support with full MERGE INTO against Iceberg and bucket and truncate partition transforms.

Querying an Iceberg table from DuckDB requires only loading the extension and pointing a scan at the table. The engine then applies projection and predicate pushdown to the underlying Parquet files.

INSTALL iceberg;
LOAD iceberg;

-- Attach a REST catalog, then scan an Iceberg table directly
ATTACH 'warehouse' AS lake (
  TYPE iceberg,
  ENDPOINT 'https://catalog.example.com/api/catalog'
);

SELECT region, sum(amount) AS revenue
FROM lake.sales.orders
WHERE order_date >= DATE '2026-07-01'   -- predicate pushed to file pruning
GROUP BY region;                        -- only 3 columns read (projection)

This model is well matched to a specific set of situations: a developer’s laptop, a notebook exploring a dataset, a continuous-integration job validating a data transformation, a small serverless function, and small-to-medium datasets generally. In each case the value is the absence of a cluster to provision and pay for. DuckDB can query an Iceberg or Delta table sitting in S3 directly, from a script, with no distributed infrastructure in between. It is not a replacement for a distributed engine on terabyte-scale joins, because a single machine cannot shuffle data across a fleet, but for a large share of everyday analytical questions the data fits and the cluster is unnecessary.

DuckLake and the Metadata-in-a-Database Shift

DuckLake is a lakehouse format that relocates where table metadata lives. In the prevailing design, exemplified by Iceberg, a table’s metadata — its snapshots, manifests, and file lists — is stored as a chain of files on the same object storage as the data. DuckLake instead stores all of that lakehouse metadata in a SQL catalog database, which may be SQLite, PostgreSQL, or DuckDB, while the data files remain Parquet on object storage (ducklake.select/2026/04/13/ducklake-10; ducklake.select/faq, as of 2026-07-11). The motivation is that many metadata operations — listing snapshots, resolving the current table version, planning which files to read — are exactly the transactional lookups that relational databases perform efficiently, whereas doing them through a tree of small files on object storage incurs many round trips.

DuckLake v1.0, described as production-ready with a backward-compatibility guarantee, was released on 2026-04-13, with a v1.1 specification expected in September 2026 (ducklake.select/2026/04/13/ducklake-10, as of 2026-07-11). Notably, the interoperability path runs through Iceberg: DuckDB’s Iceberg extension can perform a metadata-only copy of Iceberg metadata into DuckLake, after which the same underlying data files can be queried as DuckLake tables. DuckLake is treated here as a supporting anchor rather than a fourth peer engine, because its significance is architectural: it illustrates a broader movement of the catalog boundary from files on object storage into a database, which reshapes how any engine discovers and plans over lakehouse tables.

Pushdown and Caching: How Engines Avoid Reading Data

All three engines share the same fundamental challenge — object storage is slow relative to local memory — and their answers, though implemented differently, follow the same two principles. The first is to avoid reading data that a query cannot use, through pushdown. The second is to avoid re-reading data that has already been fetched, through caching. Understanding these two mechanisms clarifies why cache-warm and cache-cold performance can differ by a large margin, and why an engine’s steady-state behavior matters more than its first-query latency.

The Pushdown Flow

Pushdown proceeds in layers, and each layer that eliminates data spares every layer below it. A query’s filter first prunes partitions using the table format’s partition metadata, so that whole directories of files are skipped. Within the surviving partitions, file-level statistics eliminate individual data files whose value ranges cannot match the filter. Within a file that must be opened, Parquet’s row-group statistics allow the reader to skip row groups, and projection pushdown ensures that only the referenced columns are decoded. What reaches the engine’s operators is therefore a small fraction of the table. The detail of what these statistics contain, and how row groups and encodings are laid out, belongs to the Parquet and Arrow internals discussion; the point here is that all three engines depend on this same cascade, which is why the table and file format the data lands in — often the output of a pipeline such as the one described in the InfluxDB-to-Iceberg data pipeline — directly governs how effective any engine’s pushdown can be.

Caching Layers Compared

Pushdown reduces how much data a query reads; caching reduces how often that reading crosses the network. The three engines place a cache in front of object storage in structurally similar ways but with different scopes. Trino uses a native file-system cache for its Delta Lake, Hive, and Iceberg connectors, and can front object storage with Alluxio as a distributed cache. StarRocks provides a block-level Data Cache in its Compute Nodes and Backends, complemented in 4.0 by metadata caching that reduces cloud API calls, and its shared-data mode is explicitly designed to keep a warm local cache over remote data. DuckDB, running in a single process, benefits from operating-system page caching and its own buffer management, and reads through native extensions tuned for memory efficiency, but it has no distributed cache tier because it has no distribution.

Choosing an Engine

The three engines map cleanly onto three families of workload, and the selection is usually determined by the shape of the workload rather than by raw speed. The following table summarizes the architectural properties that drive the decision. All performance characterizations are qualitative, in keeping with the benchmark caveat stated below.

When Each Engine Fits

Trino is the appropriate choice when the requirement is heterogeneous, federated ad-hoc SQL across many sources: exploring a data lake, or standing up a shared query gateway over Iceberg, Delta, and Hive tables alongside relational and NoSQL catalogs. Its storage and compute are fully decoupled, and it scales by adding workers. It is weaker as an always-on, low-latency serving layer unless paired with a warm file-system or Alluxio cache, so a team that needs a single engine for wide-ranging exploration across systems is its natural user.

StarRocks is the appropriate choice for interactive, sub-second BI and dashboards on the lakehouse, particularly under high concurrency. Its cost-based optimizer, materialized views, and Data Cache accelerate repeated queries over Iceberg and Hive tables, and its shared-data mode preserves storage-compute separation while keeping a warm local cache. A team serving many concurrent dashboard users from lakehouse tables, where the same aggregates recur, is its natural user. Interactive serving is a different problem from batch pipelines; the boundary between low-latency serving engines and batch processing is examined in the guide to streaming versus batch data-processing architectures.

DuckDB is the appropriate choice for single-node embedded analytics: developer laptops, notebooks, continuous-integration jobs, small-to-medium datasets, and serverless functions that need to query Iceberg, Delta, or DuckLake tables directly without a cluster. It is bounded by one machine’s memory and cores and performs no distributed shuffle, so it is not a candidate for very large distributed joins, but for the large fraction of analytical work that fits on one machine it removes the operational cost of a cluster entirely.

These regions are not mutually exclusive in practice. A single organization commonly runs more than one: DuckDB in development and CI, Trino as a federated exploration gateway, and StarRocks behind the production dashboards. Because all three read the same lakehouse tables, using several is a matter of pointing each at the same object storage rather than maintaining separate copies of the data, which is precisely the flexibility the storage-compute separation was intended to provide.

Frequently Asked Questions

Can Trino, StarRocks, and DuckDB query the same Iceberg tables at once?

Yes. Because a lakehouse separates storage from compute, the Iceberg tables live in object storage independently of any engine, and all three can read them concurrently. Trino uses its Iceberg connector, StarRocks uses an external catalog, and DuckDB uses its native Iceberg extension, optionally resolving tables through a shared REST catalog. No engine owns the data, so adding one is a configuration change rather than a data migration.

Is DuckDB a replacement for Trino or StarRocks?

Not for distributed workloads. DuckDB runs in a single process and is bounded by one machine’s memory and cores, with no distributed shuffle, so it cannot spread a large join across a cluster the way Trino or StarRocks can. For datasets that fit on one machine — a common case in development, notebooks, and continuous integration — it removes the need for a cluster entirely and queries lakehouse tables directly, which is a genuine substitute for a cluster in those situations but not in large distributed ones.

Why does the same query run much faster the second time?

Because of caching. On the first execution, an engine fetches data files from object storage across the network. Trino can retain them in its file-system or Alluxio cache, StarRocks in its block-level Data Cache, and DuckDB in process and operating-system buffers. Subsequent queries that touch the same files read them from the local cache instead of object storage, which is substantially faster. This is why a benchmark must state whether the cache was warm or cold; cold-start latency is not representative of steady-state behavior.

What problem does DuckLake solve compared with Iceberg?

DuckLake stores lakehouse metadata — snapshots, manifests, and file lists — in a SQL catalog database such as SQLite, PostgreSQL, or DuckDB, rather than as a chain of small files on object storage, while keeping the data files as Parquet on object storage. Metadata operations such as resolving the current table version become transactional database lookups instead of multiple object-storage round trips. DuckLake v1.0, released on 2026-04-13, is described as production-ready, and Iceberg metadata can be copied into DuckLake so the same data files are queryable as DuckLake tables.

How do materialized views make StarRocks fast on the lakehouse?

A materialized view is a precomputed, stored result of a query. StarRocks supports partition-aligned materialized views over Iceberg and Hive tables that refresh incrementally, recomputing only the changed partitions, and its cost-based optimizer transparently rewrites an incoming query onto an eligible view without the author referencing it. A recurring scan-and-aggregate becomes a lookup. The vendor reports large speedups on eligible queries, by an order of magnitude or more, which is a self-reported and workload-dependent figure.

References

• Trino. Trino Concepts: Architecture, Coordinator, Workers, Connectors, and Catalogs. Trino Software Foundation, as of 2026-07-11.
• Trino. File System Caching for Object Storage Connectors and Alluxio file-system support. Trino Software Foundation, as of 2026-07-11.
• StarRocks. StarRocks Architecture: Frontend, Backend, and Compute Nodes. Linux Foundation, as of 2026-07-11.
• StarRocks. StarRocks 2025 Year in Review (4.0 release, materialized views, Data Cache). StarRocks, 2025 (vendor self-reported figures), as of 2026-07-11.
• DuckDB. Lakehouse Formats: Iceberg, Delta, DuckLake Extensions and New Iceberg Features in v1.5.3. DuckDB, as of 2026-07-11.
• DuckLake. Announcing DuckLake v1.0. DuckLake, 2026-04-13, as of 2026-07-11.

Conclusion

Trino, StarRocks, and DuckDB are not three answers to the same question but three answers to three different questions, unified only by the lakehouse substrate they read. Trino’s coordinator-and-worker MPP architecture and its many-catalog federation make it a strong general engine for ad-hoc SQL across heterogeneous sources, provided a caching layer is added when low latency matters. StarRocks’ frontend, backend, and compute-node design, combined with a cost-based optimizer, transparently rewritten materialized views, and a block-level Data Cache, targets interactive high-concurrency serving on the lakehouse, with a shared-data mode that separates storage from compute while retaining a warm local cache. DuckDB’s single-node, in-process model, with native Iceberg, Delta, and DuckLake extensions, brings lakehouse analytics to a laptop, a notebook, or a serverless function without any cluster, at the cost of being bounded by one machine.

The most durable guidance is to treat the engine as a component chosen per workload rather than a platform-wide commitment. Because storage and compute are separated, an organization can run all three over the same tables, matching the engine to the shape of each workload — embedded, federated, or interactive — rather than forcing every query through one engine. The emergence of DuckLake, which moves lakehouse metadata into a SQL catalog database, is a reminder that even the boundary between engine and catalog is still being redrawn, and that the compute tier of the lakehouse remains one of the more active areas of data-engineering design. An engineer who understands each engine’s architecture and its cache and pushdown behavior, rather than a single benchmark number, is positioned to make that selection correctly as the tooling continues to evolve.

Every data platform eventually confronts a single design question that shapes its cost, its latency, and its operational burden: should a given computation run over a finite dataset on a schedule, or continuously over an endless flow of events? Batch processing computes over bounded data in periodic runs, while stream processing computes incrementally over an unbounded sequence of events as they arrive. The two paradigms are often presented as rivals, but they are better understood as points on a single continuum that trades data freshness against cost, throughput, and correctness machinery. This guide provides a mechanistic, vendor-neutral model of that continuum for engineers who already operate pipelines and now need a rigorous basis for deciding where each new workload belongs.

The discussion covers the batch-versus-streaming trade-off surface, the Lambda and Kappa architecture debate, the execution split between micro-batch and event-at-a-time engines, the correctness semantics that make streaming trustworthy (event time, watermarks, windowing, and exactly-once processing), an honest account of where batch remains the correct choice, and the convergence of the two paradigms observed across 2026 toward unified engines and the streaming lakehouse.

Two Ways to Compute Over Data

The distinction between batch and streaming begins with the shape of the input. A bounded dataset is finite and complete: yesterday’s transactions, a snapshot of a table, a directory of log files. An unbounded dataset is an endless sequence of events that has no defined end, such as clickstream events, sensor readings, or a change-data-capture feed from a production database. Batch processing operates over bounded data in scheduled runs, producing a result after it has observed the entire input. Stream processing operates over unbounded data continuously, updating results incrementally as each event arrives.

This difference in input shape propagates into every operational property of a pipeline. The most useful way to reason about the choice is as a trade-off surface with three axes: latency, throughput, and cost. Latency is the delay between an event occurring and its effect appearing in a result. Throughput is the volume of records the system processes per unit of time. Cost is the compute and operational expense of running the system. These three cannot be optimized independently. Lowering latency toward the millisecond range generally requires always-on compute and additional correctness machinery, which raises both cost and operational complexity. Tolerating higher latency allows work to be amortized into scheduled bursts, which lowers steady-state cost and simplifies recovery.

A central insight, formalized by the Google Dataflow model (Akidau et al., The Dataflow Model, PVLDB 2015), dissolves the apparent opposition between the two paradigms. In that framework, a batch job is simply a stream computation over a bounded input assigned to a single global window that closes when the input is exhausted. Streaming generalizes batch rather than replacing it: the same logical operations (filtering, joining, aggregating) apply in both cases, and the differences reduce to when results are emitted and how completeness is judged. This reframing is more than a rhetorical convenience, because it underpins the unified engines discussed later, in which one API expresses both bounded and unbounded computation.

The following table summarizes how the paradigm choice affects the operational properties an engineer must plan around.

Lambda and Kappa: The Architecture Debate

Once an organization needs both historical accuracy and low-latency views, it confronts an architectural question that predates modern unified engines. The two canonical answers are the Lambda architecture and the Kappa architecture, and understanding their motivations clarifies design decisions that remain relevant even where neither is adopted by name.

The Lambda Architecture

The Lambda architecture was described by Nathan Marz, the creator of Apache Storm, around 2011 and later formalized with James Warren in the book Big Data (Manning, 2015). It composes three layers. The batch layer holds an immutable master dataset and precomputes comprehensive batch views over all historical data, prioritizing accuracy and completeness. The speed layer processes only recent data with low latency, producing approximate or incremental real-time views that compensate for the batch layer’s delay. The serving layer indexes and merges the outputs of both layers so that a query sees a combined result: authoritative history from the batch layer plus the most recent events from the speed layer.

The Lambda architecture achieves both accuracy and freshness, but at a well-known cost: the same business logic must be implemented twice, once in a batch engine and once in a stream engine, and the two implementations must be kept semantically equivalent. Any divergence between them produces inconsistent results at the serving layer, and every change to the computation must be applied and validated in both places. This dual-codebase reconciliation burden is the defining pain point of the pattern.

The Kappa Architecture

Jay Kreps, a co-creator of Apache Kafka, proposed an alternative in the 2014 essay Questioning the Lambda Architecture (O’Reilly Radar). The argument is direct: if a stream processor is sufficiently expressive and reliable, the separate batch layer is redundant. The Kappa architecture keeps a single streaming layer and backs it with a durable, replayable log, typically Kafka, that retains the raw event history. There are no longer two code paths. When the computation logic changes, or a bug is fixed, “reprocessing” means starting a new instance of the streaming job from the beginning of the retained log and letting it recompute the output, then switching consumers to the new result. The nightly recompute of the batch layer is replaced by a replay of the same code that serves live traffic.

The Kappa architecture depends on two properties of its log. The log must be durable and retained long enough to replay whatever history a recomputation requires, and it must preserve ordering per partition so that replay is deterministic. Kafka provides both, which is why it sits at the center of most Kappa deployments. Implementing a reliable consumer over that log is itself a substantial task; the mechanics of offset management, consumer groups, and rebalancing are treated in this guide to implementing a Kafka consumer in Python. The events feeding such a log frequently originate from operational databases through change-data-capture, in which row-level changes are streamed as events; the change-data-capture pattern with Debezium and Kafka is a canonical source that makes Kappa practical for database-derived data.

The industry has drifted toward Kappa because a single codebase is cheaper to maintain and because modern stream engines are expressive enough to carry the full computation. Lambda has not disappeared, however. It survives in domains where an independent, authoritative batch recomputation over an immutable master dataset serves as a correctness audit and a reconciliation baseline, particularly where regulatory or financial reporting demands a reproducible ground truth computed separately from the live path. In practice, many teams run a hybrid: a Kappa-style streaming path for freshness and a periodic batch job that reconciles and corrects, which is closer to Lambda in spirit than either label admits.

Execution Models: Micro-Batch and Event-at-a-Time

Below the architectural layer sits a second decision that determines a pipeline’s latency floor and recovery behavior: how the engine physically executes the stream. Two designs dominate, and they differ in the unit of work they process at a time.

Micro-batch execution

Spark Structured Streaming uses micro-batch execution by default. The incoming stream is divided into a sequence of small, deterministic batches, and each batch is executed as an ordinary Spark job over the records that accumulated during a short interval. This design inherits the fault-tolerance and exactly-once guarantees of Spark’s batch engine almost for free, because recovery means re-running a deterministic batch. According to the Spark Structured Streaming Programming Guide (version 4.1.x, 2026), the default micro-batch engine achieves end-to-end latencies as low as roughly 100 milliseconds with exactly-once guarantees. The trade-off is that the batch boundary imposes a latency floor: a result cannot appear until its micro-batch has been formed and executed. Micro-batch execution is well suited to high-throughput workloads where a latency of a fraction of a second is acceptable.

Spark has offered lower-latency options over time. A Continuous Processing mode, introduced experimentally in Spark 2.3, processes records with latency near one millisecond but provides only at-least-once guarantees. More recently, Spark 4.1 added a Real-Time Mode for Structured Streaming that targets sub-second and single-digit-millisecond latency for stateless tasks (Spark 4.1.0 release notes and the associated Databricks engineering blog, 2026).

Event-at-a-time execution

Apache Flink processes each record as it arrives, one event at a time, through a long-lived graph of operators. There is no batch boundary to wait for, so latency is bounded by the processing of an individual record rather than by an interval. This event-at-a-time model gives Flink native event-time semantics and a rich model of state and timers, which makes it a natural fit for complex, low-latency stream computations. Pattern-matching over event streams is one such application; the complex event processing pipeline with Flink CEP illustrates how the event-at-a-time model supports detecting temporal patterns across a continuous stream.

Apache Beam occupies a distinct position in this landscape. Rather than being an execution engine, Beam is a programming model that implements the Dataflow abstraction and compiles the same pipeline to different execution engines, called runners, including Flink and Spark. The Beam release line reached version 2.74.0 (2026-06-02), which added a Spark 4 runner for the Java software development kit (Beam downloads page, 2026). Beam lets an engineer write a computation once against the Dataflow model and choose the execution backend separately, which is the clearest embodiment of the “batch is a special case of streaming” principle.

The engine version landscape as of mid-2026 is worth noting for teams selecting a platform. Apache Flink reached 2.3.0 (2026-06-25), with a long-term-support line at 1.20.x, following the 2.0.0 major release in March 2025. Apache Spark’s current stable releases are 4.1.2 (2026-05-21) and 4.0.3 on the 4.0 line. Apache Kafka reached 4.3.1 (2026-06-25); notably, Kafka 4.0.0 removed ZooKeeper entirely, making the KRaft consensus protocol the only operating mode (Flink, Spark, and Kafka download and release pages, 2026).

Getting Streaming Correct: Time, Watermarks, Windows, and Semantics

The difficulty of streaming is not moving events quickly; it is producing correct answers over data that arrives late, out of order, and without a natural end. This section covers the machinery that makes streaming results trustworthy, which is the substance that separates a robust pipeline from one that silently produces wrong aggregates.

Event time versus processing time

Event time is the moment an event actually occurred, recorded as a timestamp embedded in the record itself. Processing time is the moment the pipeline observes the event, which depends on network delay, buffering, and system load. In any real distributed system these two clocks diverge, and the gap between them, called event-time skew, is variable. A mobile device that loses connectivity may deliver an event minutes after it occurred. Because business questions are almost always framed in terms of when events happened rather than when a server saw them, correct windowed aggregation must be computed in event time. Aggregating in processing time is simpler but produces results that shift depending on system latency, which makes them non-reproducible.

Watermarks

A watermark is a monotonic assertion about completeness in the event-time domain. A watermark of value T declares that the system expects no further events with a timestamp at or before T. This is precisely the mechanism a streaming system uses to decide when an event-time window is complete enough to close and emit a result. A watermark that advances aggressively produces low-latency results but risks excluding genuinely late events; a conservative watermark waits longer, includes more late data, and increases latency. The watermark is therefore the tuning knob that trades completeness against latency, and it is defined per pipeline based on how late data is expected to arrive.

Events that arrive after the watermark has passed their window are late data. Systems handle late data through configurable policies: an allowed-lateness period keeps a window’s state alive for a grace interval, triggers can fire refined results as more data arrives, and retractions can withdraw and correct a previously emitted result. These options are drawn directly from the Dataflow model’s framing of when results are materialized and how refinements relate to earlier outputs.

Windowing

Windowing divides an unbounded stream into finite chunks over which aggregation is defined. Three window types are canonical. A tumbling window is fixed-size, contiguous, and non-overlapping, so each event belongs to exactly one window; a five-minute tumbling window partitions time into adjacent five-minute blocks. A sliding window has a fixed size and a separate slide interval and therefore overlaps, so a single event can belong to several windows; a ten-minute window that advances every minute is a sliding window. A session window is data-driven and gap-based: it groups events separated by less than a configured inactivity gap and closes after the gap elapses, producing variable-length windows that are not aligned across keys, which is well suited to modeling bursts of user activity.

Delivery semantics and exactly-once

Delivery semantics describe how many times an event’s effect can be reflected in the output when failures and retries occur. At-most-once processing may drop records and never duplicates them; it is fire-and-forget and cheapest. At-least-once processing never loses a record but may apply it more than once after a retry or recovery, producing duplicates. Exactly-once processing guarantees that each record affects the computed state exactly once despite failures, which is what most correctness-sensitive aggregations require.

Flink achieves exactly-once state consistency through asynchronous barrier snapshotting, a variant of the Chandy-Lamport distributed snapshot algorithm (Flink Stateful Stream Processing documentation, 2026). The coordinating JobManager periodically injects special records called checkpoint barriers into the streams at the sources. As a barrier flows downstream, it separates the records that belong to the current snapshot from those that belong to the next one. When an operator has multiple input channels, it performs barrier alignment: it waits until the barrier has arrived on every input channel before taking its snapshot, buffering records that arrive after the barrier on faster channels. Once all operators have snapshotted their state, the checkpoint is complete and can be used to restore the entire job after a failure.

Internal exactly-once state is not sufficient by itself. To make results exactly-once all the way to an external system, the sink must participate in the checkpoint through a two-phase commit: it prepares (writes data in an uncommitted transaction) as part of a checkpoint and commits only after the checkpoint has completed successfully, so that a failure before completion leaves nothing visible to downstream readers. Flink implements this through a transactional committer for sinks that support transactions or idempotent writes. Kafka’s transactional producer and its durable, ordered-per-partition log are what make such end-to-end guarantees possible on the sink side.

Barrier alignment has a cost. Under backpressure, an operator can stall while waiting for a barrier on a slow channel, which delays the checkpoint and raises latency. This motivated unaligned checkpoints, in which barriers overtake buffered records and the in-flight data is included in the snapshot instead, trading a larger checkpoint for shorter alignment delay. The default in Flink remains exactly-once with aligned checkpoints, and the choice to relax it is a deliberate latency optimization.

When Batch Remains the Right Choice, and the Convergence

The correctness machinery above should make one point clear: streaming buys freshness at the price of real complexity. For many workloads that price is not justified, and batch remains the correct and often superior choice. Recognizing these cases is as important as knowing how to build a stream.

Batch is the better fit for large historical backfills, where terabytes of accumulated data must be reprocessed and there is no latency requirement at all. It is preferable for full-refresh reproducibility, where the ability to re-run a deterministic job and obtain an identical, auditable result is worth more than freshness. It suits complex multi-source joins that tolerate latency, cost-sensitive periodic reporting where always-on compute would be wasteful, and the construction of machine-learning training sets, which are inherently snapshots over a bounded, versioned dataset. It also remains the natural home for correctness-audit reprocessing, the independent recomputation that underpins the surviving uses of Lambda.

The batch layer has its own mature tooling. Scheduled batch pipelines are commonly orchestrated with a workflow scheduler; the Apache Airflow orchestration guide describes how directed acyclic graphs express batch dependencies and retries. Transformation logic over warehouse tables is frequently expressed with a batch-first tool as covered in the dbt transformation pipeline guide. Batch outputs typically land in columnar files whose layout is examined in the Parquet and Arrow internals guide, and interactive analysis over such files increasingly runs in in-process engines compared in the DuckDB and Polars comparison.

The 2026 convergence

The sharp dichotomy between batch and streaming is eroding along three lines. The first is unified engines and APIs: Beam’s single model runs on multiple runners, and both Spark and Flink execute bounded and unbounded jobs through largely shared machinery, so the same logic serves both modes. The second is incremental processing: rather than fully reloading a dataset, a pipeline recomputes only the portion that changed, which imports a streaming efficiency into what looks like a batch job. The third is the streaming lakehouse, in which a single table serves both batch queries and streaming reads and writes.

Open table formats provide the foundation for the lakehouse by adding transactional guarantees and metadata to files in object storage; the trade-offs among them are examined in the Iceberg, Delta Lake, and Hudi comparison. A streaming-native format, Apache Paimon, an Apache Top-Level Project since 2024, extends this idea further by combining a log-structured merge-tree with a changelog stream, so that the same table can be written and read as a continuous stream while also supporting batch queries and incremental reads (Apache Paimon project, 2026). The practical effect is that Lambda’s two separate stores collapse into one, which removes the reconciliation burden that motivated the Kappa argument in the first place.

A Decision Checklist

The following questions guide a workload toward batch or streaming without prescribing a specific engine. They are ordered so that the strongest determinants come first.

When several answers point in the same direction, the decision is clear. When they conflict, the convergence tooling offers a middle path: build on a unified engine and a lakehouse table so that a workload can begin as batch and gain a streaming read later without a rewrite, deferring the commitment until the freshness requirement is genuinely established.

Frequently Asked Questions

Is streaming always more expensive than batch?

Not in a fixed ratio. Streaming runs always-on compute, which produces a steady cost that pays for freshness, while batch runs in scheduled bursts whose cost is amortized. Whether streaming costs more for a given workload depends on the state size, the shuffle behavior, and how continuously the data actually arrives. It is more accurate to reason about the shape of the trade-off, always-on versus bursty, than to apply a single multiplier.

Why not just use processing time instead of event time?

Processing-time aggregation is simpler because it ignores when events actually occurred, but its results shift with system latency and are therefore non-reproducible. If a network delay causes events to arrive late, a processing-time window attributes them to the wrong interval. Business questions are almost always framed in event time, so correct windowed results require it, which in turn requires watermarks to judge completeness.

What is the practical difference between at-least-once and exactly-once?

At-least-once never loses a record but may apply it more than once after a failure, producing duplicates. Exactly-once guarantees each record affects the result exactly once, at the cost of checkpointing and, for external sinks, a two-phase commit. If the downstream system deduplicates or the operation is idempotent, at-least-once can deliver equivalent correctness more cheaply; exactly-once is warranted when duplicates genuinely corrupt the output.

Does the convergence toward unified engines make Lambda and Kappa obsolete?

It reduces the cost that motivated the debate rather than settling it. Streaming lakehouse table formats collapse Lambda’s two stores into one, which removes much of the reconciliation burden. An independent batch recomputation still has value as a correctness audit in regulated or reconciliation-heavy domains, so Lambda-style patterns persist even as the tooling makes a single code path easier to maintain.

Is micro-batch a form of batch or streaming?

It is streaming implemented by repeatedly running very small batch jobs. Spark Structured Streaming chops an unbounded stream into short, deterministic micro-batches, which inherits batch-style recovery and exactly-once guarantees while continuously producing results. It sits between the two paradigms and illustrates why the batch-streaming boundary is better viewed as a continuum than a hard line.

References

• Akidau, T., et al. The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing. PVLDB 8(12), 2015.
• Kreps, J. Questioning the Lambda Architecture. O’Reilly Radar, 2014.
• Apache Flink. Stateful Stream Processing: Checkpoints, Barriers, and Exactly-Once. Apache Software Foundation, 2026.
• Apache Spark. Structured Streaming Programming Guide. Apache Software Foundation, 2026.
• Apache Beam. Beam Programming Guide and Downloads. Apache Software Foundation, 2026.
• Apache Paimon. Apache Paimon: Streaming Lakehouse Table Format. Apache Software Foundation, 2026.

Conclusion

Batch and stream processing are not opposing technologies but positions on a single trade-off surface defined by latency, throughput, and cost. The Dataflow model makes this concrete by treating a batch job as a stream computation over a bounded input in a single global window, which is why the same engines and APIs increasingly express both. The architectural debate between Lambda and Kappa reduces to whether an independent batch recomputation is worth its dual-codebase burden, and the industry has drifted toward the single-path Kappa answer while retaining Lambda where reconciliation demands it. Beneath the architecture, the choice of execution model, micro-batch or event-at-a-time, sets a pipeline’s latency floor and recovery behavior, and the correctness machinery of event time, watermarks, windowing, and exactly-once processing is what makes streaming results trustworthy rather than merely fast.

The most durable guidance is to resist treating either paradigm as a default. A workload should be placed on the continuum according to how fresh its result must be and what that freshness is worth, and the 2026 convergence toward unified engines and streaming lakehouse tables increasingly allows that decision to be deferred and revised without a rewrite. An engineer who understands the trade-off surface, rather than memorizing a preferred stack, is equipped to make that judgment correctly for each new dataset.

Modern data platforms move records across many independent systems: an application emits an event, an ingestion job lands it in a lake, a transformation framework reshapes it, and a dashboard or a machine-learning model consumes the result. Every one of these handoffs is an implicit agreement about the shape and meaning of the data, and every silent violation of that agreement propagates downstream until a report is wrong or a model degrades. A data contract—a versioned, machine-readable agreement between the team that produces a dataset and the teams that consume it, specifying its schema, semantics, service-level guarantees, and quality expectations—turns that implicit agreement into an explicit, testable artifact. This post examines how data contracts and data quality enforcement work together to make pipelines reliable, and why the two ideas are complementary rather than interchangeable.

The distinction that organizes the discussion is between prevention and detection. A data contract prevents bad data from entering a system by asserting expectations at the boundary where data is produced. Data observability, by contrast, detects problems after data has already landed, by monitoring freshness, volume, and schema drift and raising anomaly alerts. A mature platform uses both. The material below defines the contract as an artifact, presents a taxonomy of six data quality dimensions, surveys the enforcement engines that operate at the record, DataFrame, warehouse, and stream layers, and describes the patterns—shift-left validation, continuous-integration gates, pre-ingestion quarantine, circuit breakers, and dead-letter queues—that put a contract into force.

The Data Contract as a First-Class Artifact

A data contract is a formal, versioned agreement between a data producer and its consumers. The producer is the system or team that emits a dataset—an application service publishing events, a database exposing a table, or an ingestion job writing files. The consumers are the downstream systems and teams that read that dataset: analytics models, dashboards, machine-learning features, and other services. Before contracts, the terms of that relationship lived in tribal knowledge and out-of-date documentation, so a producer could rename a column or change a unit of measurement without warning, and consumers discovered the break only when their output failed. A contract makes the terms explicit and machine-readable, so that a change which would violate them can be caught automatically.

Four elements typically appear in a contract. The schema declares the fields, their types, and their nullability. The semantics describe what each field means, including units, allowed value sets, and business definitions, so that a field named revenue is unambiguous about currency and whether it is gross or net. The service-level agreement (SLA) states operational guarantees such as freshness, expected update frequency, and availability. The quality expectations encode testable rules—for example, that a primary key is unique or that a percentage column falls between zero and one hundred. Figure 1 shows how these four elements sit on the boundary between a producer and its consumers.

A Standard for the Contract: ODCS

Writing a contract in an ad-hoc format leaves every team to invent its own structure, so a shared specification is useful. The Open Data Contract Standard (ODCS) is a vendor-neutral YAML format that describes the agreement between a data producer and its consumers. Its current version is v3.1.0 (Bitol / Linux Foundation AI & Data, as of 2026). ODCS originated as PayPal’s internal data contract template, was open-sourced, and was donated to the Linux Foundation; it is now stewarded by Bitol, a Linux Foundation AI & Data incubation project licensed under Apache 2.0 that was formed on 30 November 2023 when the AIDA User Group and LF AI & Data joined forces (bitol.io, as of 2026). Bitol also stewards the Open Data Product Standard (ODPS), which has reached v1.0.0.

An important point about the current landscape is that the ecosystem is consolidating around a single standard. A separate effort, the Data Contract Specification, is being deprecated and is converging on ODCS v3.1.0. Its maintainers advise users to migrate, and tooling support in the Data Contract CLI and Entropy Data is planned only until the end of 2026 (datacontract-specification.com, as of 2026). A team beginning with contracts in 2026 should therefore standardize on ODCS rather than the older specification. A minimal ODCS-style contract illustrates the artifact concretely.

apiVersion: v3.1.0
kind: DataContract
id: orders-contract
name: Orders
version: 1.2.0
status: active
schema:
  - name: orders
    properties:
      - name: order_id
        logicalType: string
        required: true
        unique: true
      - name: customer_id
        logicalType: string
        required: true
      - name: order_total
        logicalType: number
        required: true
        quality:
          - rule: minimum
            mustBeGreaterThanOrEqualTo: 0
      - name: status
        logicalType: string
        required: true
        quality:
          - rule: enum
            values: [active, shipped, cancelled]
slaProperties:
  - property: freshness
    value: 15
    unit: minute

Treating the contract as versioned code has a direct engineering consequence: a change to the contract is a change to a file in version control, subject to review and to automated compatibility checks. This is what makes the shift-left patterns described later in this post possible, and it links data contracts to the broader practice of treating transformation logic as code, as covered in the guide to dbt for building transformation pipelines.

Six Dimensions of Data Quality

Before a contract can assert quality expectations, quality itself must be defined precisely enough to be measured. A widely used approach decomposes quality into distinct dimensions, each answering a different question about the data. The six dimensions below form a compact and practical taxonomy. Each is defined on first use, because the terms are often used loosely in practice.

Completeness asks whether the expected data is present, with no missing values or absent records where they are required. A completeness check might assert that customer_id is never null, or that a daily table’s row count falls within an expected band. Uniqueness asks whether each real-world entity appears once, with no unintended duplicates; a typical check enforces primary-key uniqueness so that no order_id is repeated. Validity asks whether values conform to a defined format, type, range, or allowed set—the domain of the field. Examples include requiring that status belongs to a fixed set of allowed values, that an email address matches a regular expression, or that an age is not negative.

Accuracy asks whether values correctly describe the real-world entity they represent. This is among the hardest dimensions to test in isolation, because it requires comparison against a trusted system of record rather than an internal rule; a value can be valid and unique yet still wrong. Consistency asks whether values agree across systems, tables, and over time, with no contradictions—for instance, that the sum of an order’s line items equals its recorded total, or that a customer’s country is identical in every table. Timeliness asks whether data is fresh and available within its expected latency or SLA window; a timeliness check verifies that the maximum event timestamp is within the last few minutes or that a partition landed on schedule.

Accuracy and consistency are the two dimensions that resist mechanical enforcement most strongly, because they depend on external reference points and cross-system reconciliation rather than a self-contained rule. A contract can assert them, but verifying them often requires a comparison the pipeline cannot perform on its own. A seventh dimension, integrity—referential integrity, meaning that foreign keys resolve to existing rows—is sometimes added; it can be treated as a specialized form of consistency across tables.

Where Enforcement Happens: Four Layers and Their Engines

A single contract is enforced at several points, because data changes form as it moves. It arrives as individual records at an application boundary, is assembled into in-memory tables (DataFrames) for processing, is materialized as warehouse tables, and travels as messages on streams. Each form has a matching class of validation engine. The essential distinction—already stated for the contract itself—applies here too: the specification declares what must hold, and each engine enforces it in the representation it understands.

Record and DataFrame Boundaries

At the application boundary, data arrives one record at a time—an incoming API request, a single event to be published. Pydantic validates and parses data at this record level. Its current version is v2.13.4 (pydantic/pydantic releases, as of 2026). The validation engine lives in a component called pydantic-core, written in Rust through the pyo3 binding layer, which delivers roughly a fivefold to fiftyfold performance improvement over version 1 depending on the workload (pydantic.dev, as of 2026). Pydantic excels at fast per-record validation at ingestion and API edges, but it operates one object at a time and is not designed for table-scale statistical checks.

Once records are assembled into a DataFrame—an in-memory tabular structure—the relevant unit of validation becomes the column and the table rather than the single object. Pandera provides schema-based, statistical validation for DataFrames. Its current version is 0.32.1 (unionai-oss/pandera releases, as of mid-2026). Pandera 0.19.0 added Polars validation, and by version 0.29 (January 2026) a single schema definition could validate data across pandas, Polars, Dask, Modin, PySpark, and Ibis; a Narwhals-powered backend introduced in 0.32.0 adds lazy validation across Polars, Ibis, and PySpark SQL. A Pandera schema expresses column-level expectations directly.

import pandera as pa
from pandera import Column, Check

schema = pa.DataFrameSchema({
    "order_id":    Column(str, unique=True, nullable=False),
    "customer_id": Column(str, nullable=False),
    "order_total": Column(float, Check.ge(0)),
    "status":      Column(str, Check.isin(["active", "shipped", "cancelled"])),
})

# Raises SchemaError on the first violation, or collects all
# violations when lazy=True.
validated = schema.validate(df, lazy=True)

Warehouse Tables

Inside the warehouse, checks run against materialized tables in SQL. Three engines are common here. dbt tests co-locate assertions with the transformation models that produce a table. dbt Core’s current version is 1.11.12 (dbt-core GitHub releases, as of 2026-07-01), with a Rust-based v2.0.0-alpha in development as the foundation for the dbt Fusion engine. dbt distinguishes data tests—generic and singular assertions that run against materialized data—from native unit tests, which validate SQL logic and were introduced in v1.8; in v1.9 and later, dbt test --resource-type test runs data tests while excluding unit tests. Because dbt tests only cover what dbt materializes, they do not guard the ingestion boundary or streams. The role of dbt in the transformation layer is treated fully in the guide to dbt transformation pipelines.

Great Expectations (GX Core) is a dedicated validation framework built around a reusable library of expressive assertions called Expectations, together with automatic profiling and human-readable validation reports called Data Docs. Its current version is 1.18.2 (Great Expectations changelog, as of 2026-06-26); GX Core 1.0 was a major API redesign that separated the open-source GX Core from the commercial GX Cloud. It supports SQL, Spark, and pandas backends. Its strength is the breadth and reusability of its Expectation library; historically its weakness has been a heavier setup and a steeper learning curve. Figure 5 traces its validation flow.

Soda Core is a second dedicated framework, known for concise checks and a continuous-integration-friendly command-line interface. Its current version is 4.16.0, released 29 June 2026, and it requires Python 3.10 or later (Soda Core release notes, as of 2026-06-29). A notable change is that Soda Core version 4 makes data contracts the default way to define quality rules—a breaking change that moves away from the older SodaCL “checks” syntax toward a contract-based syntax. A concise Soda check reads close to natural language.

checks for orders:
  - row_count > 0
  - missing_count(customer_id) = 0
  - duplicate_count(order_id) = 0
  - invalid_count(status) = 0:
      valid values: [active, shipped, cancelled]
  - freshness(order_time) < 15m

Streams

On a Kafka stream, the contract is the message schema, and the enforcement point is a schema registry—a service that stores schemas and rejects producers whose schema is incompatible with the registered version. Confluent Schema Registry supports Avro, Protobuf, and JSON Schema (referred to as JSON_SR) out of the box on both Confluent Platform and Confluent Cloud (docs.confluent.io, as of 2026). It governs schema evolution through compatibility types: BACKWARD (the default), BACKWARD_TRANSITIVE, FORWARD, FORWARD_TRANSITIVE, FULL, FULL_TRANSITIVE, and NONE. BACKWARD is the default because it allows consumers using the new schema to read data written with the previous schema, which lets a consumer rewind to the start of a topic. Confluent recommends BACKWARD_TRANSITIVE for Protobuf, because adding new message types is not forward compatible, whereas Avro was designed with schema evolution in mind (docs.confluent.io, as of 2026). A schema registry enforces the shape and evolution of messages, not their semantic quality; it will not catch a null in a field that is technically nullable or a value out of an expected range. Streaming schema enforcement pairs naturally with change-data-capture pipelines, as discussed in the guide to change data capture with Debezium and Kafka, and with the consumer-side handling covered in the Kafka consumer implementation guide.

The following table summarizes what each engine does best and where its boundary lies. Every version cited carries its source and date in the prose above.

Enforcement Patterns: Shift-Left, Gates, and Dead-Letter Queues

Having an engine is not the same as having a strategy for where and when it runs. Several established patterns place enforcement at different points in the data’s journey, trading availability against correctness in different ways.

Shift-Left at the Producer Boundary

Shift-left is the practice of moving validation as early as possible—toward the point where data is produced—rather than catching problems late in the pipeline. The term borrows from software testing, where moving tests earlier (“to the left” on a left-to-right timeline) reduces the cost of fixing defects. Applied to data, shift-left means validating at the application or ingestion boundary so that non-conforming data never enters the warehouse at all. This is the cheapest place to catch an error, because the cost of a defect rises as it travels downstream: a bad value caught at the producer affects nothing, whereas the same value caught at the BI or ML layer may have already corrupted reports, retrained a model, or been copied into many derived tables. Figure 3 depicts this rising cost.

Continuous-Integration Gates on Contract Changes

Because a contract is a file under version control, a proposed change to a producer’s schema arrives as a pull request. A continuous-integration (CI) gate is an automated check that runs on that pull request and blocks the merge if the change would break the contract. Two checks matter here: validating that the contract file itself is well formed, and running a backward-compatibility check that determines whether existing consumers can still read data produced under the new schema. This mirrors exactly what a schema registry does at runtime for streams, moved earlier to the moment of code review. A breaking change—removing a required field, narrowing a type—fails the gate and cannot merge until the producer and consumers agree on a migration. Figure 6 shows the contract advancing through versions with a compatibility check at each transition.

Pre-Ingestion Gates and Circuit Breakers

When validation cannot happen strictly at the producer—because the producer is a third party, or the data arrives as files—a pre-ingestion gate places incoming data in a staging area first and requires a validation step to pass before the data is promoted to a production table. The gate fails closed: if the check does not pass, the data does not advance. This quarantines suspect data rather than exposing it to consumers.

A related pattern is the circuit breaker, which halts a running pipeline when a contract check fails beyond a chosen threshold, rather than allowing bad data to propagate. The name is borrowed from electrical engineering, where a breaker interrupts a circuit to prevent damage. In a data pipeline hosted by an orchestrator, a failed check fails the orchestrator task and stops downstream jobs from running. This deliberately trades availability for correctness: a stalled pipeline is often preferable to a fast one that delivers wrong answers, because a visible delay prompts investigation whereas a silent error can persist unnoticed for days. Orchestrators such as those covered in the Apache Airflow orchestration guide are the natural host for both pre-ingestion gates and circuit breakers, because they already model tasks, dependencies, and failure propagation. A validation task placed upstream of a load task allows the orchestrator to skip or fail the load automatically when the check does not pass, so the circuit-breaker behavior falls out of the dependency graph rather than requiring bespoke error handling.

Dead-Letter Queues for Streaming

In a streaming pipeline, halting the entire stream because a single record fails a check would be too blunt, since one malformed message would block every well-formed one behind it. A dead-letter queue (DLQ) solves this by routing records that fail schema or quality checks to a separate queue for inspection and later replay, while valid records continue on the main stream. This isolates failures at the level of the individual record rather than the whole pipeline, and it preserves the malformed records for diagnosis instead of discarding them. Figure 4 shows a stream splitting into a pass branch and a dead-letter branch, with a replay path back into the pipeline once the underlying problem is fixed.

Contracts and Observability: Prevention Versus Detection

Data contracts are frequently discussed alongside data observability, and the two are sometimes conflated. They address the same goal—reliable data—from opposite directions. A contract is a mechanism of prevention: it asserts expectations at the boundary and refuses data that violates them, so a defect is stopped before it enters the system. Observability is a mechanism of detection: it monitors the data that has already landed—tracking freshness, volume, distribution, and schema drift—and raises an alert when reality departs from the norm. Observability finds problems the contract did not anticipate, including gradual distribution shifts and upstream failures that produce technically valid but unusual data. Figure 8 places the two side by side over a shared pipeline.

Neither mechanism substitutes for the other. A contract cannot anticipate every failure mode, particularly slow statistical drift in otherwise valid data; observability catches those. Observability, on its own, only tells a team that something has already gone wrong, often after consumers have been affected; a contract prevents the class of failures it can express. The two belong together in a mature stack. Quality gates also interact with the storage layer, because a table’s schema evolution is itself a contract concern—an area explored in the comparison of Iceberg, Delta Lake, and Hudi table formats and in the discussion of typing at the file level in the guide to Parquet and Apache Arrow internals. An end-to-end pipeline where these gates would apply in practice is described in the walkthrough from InfluxDB to AWS Iceberg with Telegraf.

The Business Case for Enforcement

The motivation for this discipline is that poor data quality carries a measurable cost. A frequently cited estimate holds that poor data quality costs organizations at least 12.9 million US dollars per year on average; this figure comes from Gartner’s 2020 Magic Quadrant for Data Quality Solutions, based on a survey of 154 reference customers across 16 vendors (Gartner, as of 2020). It should be read as a 2020 estimate rather than a current measurement, and organizations vary widely, but it captures the order of magnitude at stake. The cost is not only financial. Erroneous data erodes the trust that consumers place in a dataset, and once a dashboard has been visibly wrong, downstream teams begin to second-guess every figure it produces, which slows decisions and encourages the growth of parallel, unofficial data copies. Enforcement at the boundary is a way to protect that trust as much as it is a way to avoid direct cost. Adoption of data contracts is growing as teams formalize the producer-consumer relationship, although a precise adoption figure is not available from a reliable primary survey. The direction of travel is clear from the standards themselves: the consolidation of the Data Contract Specification into ODCS v3.1.0 under the Linux Foundation signals a maturing field converging on shared conventions.

Frequently Asked Questions

What is the difference between a data contract and a database schema?

A database schema declares field names, types, and nullability—the structural shape of a table. A data contract is broader: it wraps the schema together with semantics (what each field means, including units and allowed value sets), a service-level agreement (freshness and availability guarantees), and testable quality expectations, and it is versioned as an explicit agreement between the producing team and its consumers. In short, a schema is one component of a contract, and the contract adds meaning, guarantees, and ownership on top of structure.

Do I need both a schema registry and a tool like Great Expectations?

They cover different concerns and are often used together. A schema registry, such as Confluent Schema Registry, enforces the shape and compatible evolution of messages on a stream, rejecting a producer whose schema is incompatible with the registered version. It does not check semantic quality such as null values in nullable fields, out-of-range numbers, or freshness. A tool like Great Expectations or Soda Core enforces those quality expectations against tables or DataFrames. A streaming platform that also cares about value-level quality therefore benefits from both.

Which data quality dimensions are hardest to enforce automatically?

Accuracy and consistency are the hardest. Accuracy asks whether a value correctly describes the real-world entity it represents, which requires comparison against a trusted system of record rather than a self-contained rule; a value can be well-formed, unique, and within the allowed set yet still be wrong. Consistency asks whether values agree across systems and over time, which requires cross-system reconciliation. Completeness, uniqueness, validity, and timeliness are comparatively straightforward to express as mechanical checks.

Should a team standardize on ODCS or the Data Contract Specification in 2026?

A team beginning in 2026 should standardize on the Open Data Contract Standard (ODCS), currently at version 3.1.0 under the Linux Foundation’s Bitol project. The alternative Data Contract Specification is being deprecated and is converging on ODCS, with its tooling support in the Data Contract CLI and Entropy Data planned only until the end of 2026 (datacontract-specification.com, as of 2026). Choosing the surviving standard avoids a forced migration later.

References

• Open Data Contract Standard (ODCS) v3.1.0 — Bitol / Linux Foundation AI & Data (as of 2026)
• Great Expectations (GX Core) changelog (as of 2026-06-26)
• Soda Core release notes (as of 2026-06-29)
• Confluent Schema Registry — Schema Evolution and Compatibility (as of 2026)
• dbt data tests and unit tests documentation (as of 2026-07-01)
• Pandera documentation (as of mid-2026)

Conclusion

Data reliability is achieved not by a single tool but by a discipline that combines an explicit artifact with layered enforcement. The data contract turns the implicit producer-consumer agreement into versioned, machine-readable code, and the Open Data Contract Standard v3.1.0 provides a vendor-neutral way to express it; the consolidation of the older Data Contract Specification into ODCS marks a field settling on shared conventions. Quality itself becomes testable once it is decomposed into completeness, uniqueness, validity, accuracy, consistency, and timeliness, with the understanding that accuracy and consistency resist mechanical checks and require external reference points.

The contract is then enforced wherever data changes form: Pydantic and Pandera at the record and DataFrame boundary, dbt tests and Great Expectations and Soda Core in the warehouse, and a schema registry on the stream. The patterns that put enforcement into force—shift-left validation at the producer, continuous-integration gates on contract changes, pre-ingestion quarantine, circuit breakers, and dead-letter queues—each trade some availability for correctness in the way best suited to their layer. Finally, contracts and observability are complementary rather than interchangeable: contracts prevent the failures they can express at the boundary, while observability detects the anomalies they cannot anticipate. A platform that runs both, over a contract treated as a first-class artifact, is one whose data consumers can trust.

Apache Parquet and Apache Arrow are the two most widely deployed columnar data formats in modern analytics, yet most engineers who depend on them daily treat them as opaque. A query runs against a Parquet file through Spark, DuckDB, Polars, or pandas, the result returns quickly, and the reasons remain hidden. This post opens the black box. It describes what the bytes of a Parquet file actually look like on disk, how encoding and compression stack into a layered pipeline, how metadata and statistics let a reader skip most of the data it never needs, and how Apache Arrow provides the complementary in-memory format that lets independent engines exchange columns without paying a serialization cost. The goal is a mechanistic understanding that translates directly into tuning decisions: row-group sizing, encoding choice, compression codec, page index, and Bloom filters.

The two projects are designed to work together. Parquet is a space-efficient format optimized for durable storage on disk or in object storage. Arrow is a compute-efficient format optimized for processing in memory. A reader decodes Parquet pages into Arrow arrays, operates on them in a CPU-friendly layout, and hands them to another library through raw memory pointers. Understanding both halves, and the bridge between them, explains why a column-oriented analytical stack behaves the way it does.

Why columnar storage exists

A data file must choose an order in which to write the values of a table. A row-oriented format stores all fields of the first record together, then all fields of the second record, and so on. This layout suits transactional workloads that read or write entire records one at a time. An analytical workload behaves differently. It scans a small number of columns across a very large number of rows, computing aggregates, filters, and joins. For such a query, a row-oriented file forces the reader to touch every field of every record even when only two of forty columns are relevant.

A column-oriented, or columnar, format inverts the layout. It stores all values of the first column together, then all values of the second column, and so on. A query that reads two columns reads only those two contiguous regions and ignores the rest of the file. This property, reading only the requested columns, is called projection pushdown, and it is essentially free in a columnar format because the requested data is already grouped.

Columnar layout produces a second benefit that compounds the first. Values within a single column share a data type and often share statistical structure — a timestamp column tends to increase monotonically, a category column repeats a small set of strings, a sensor column clusters around a mean. Grouping like values together lets the format apply type-aware encodings and lets general-purpose compressors find far more redundancy than they could in an interleaved row layout. The figure below contrasts the two layouts for a small table.

This division of labor is deliberate. A durable storage format optimizes for size, because bytes on disk or in object storage cost money and network transfer time. An in-memory format optimizes for processing speed, because a CPU or GPU benefits from predictable, contiguous, aligned buffers. Apache Parquet fills the first role and Apache Arrow fills the second. The remainder of this post examines each in turn, then the bridge that connects them.

Anatomy of a Parquet file on disk

A Parquet file is a self-describing container built from a strict hierarchy. Understanding that hierarchy is the foundation for every tuning decision that follows, because each level maps to a unit of I/O or a unit of metadata.

The file, row group, column chunk, and page

At the top is the file. A file is divided horizontally into row groups. A row group is a partition of the rows — a contiguous band of records — and it is the largest unit that can be processed independently. Within a row group, each column is stored as a column chunk, which holds every value of that one column for that band of rows. Each column chunk is in turn divided into pages. A page is the smallest unit that must be read and decompressed in full to access any record inside it, and each page is encoded and compressed independently. The default row group size is 128 MB (the parquet.block.size configuration is 134217728 bytes) and the default page size is 1 MB (the parquet.page.size configuration is 1048576 bytes), according to the Parquet configuration documentation (parquet.apache.org, as of 2026-07-02).

The file is framed by a four-byte magic value, PAR1, at both the beginning and the end. The trailing region before the closing magic is the footer, which holds the FileMetaData: the schema, the list of row groups, the metadata and statistics for every column chunk, and the byte offsets that locate optional structures such as the Page Index and Bloom filters. A reader opens a Parquet file by seeking to the end, reading the footer, and using it as a map to the rest of the file. The figure below shows this layout.

The Parquet specification defines this structure in two parts that are read together: a prose format document and a Thrift definition file, parquet.thrift, that specifies the exact metadata structures (apache/parquet-format README, as of 2026-07-02). The latest format specification version is 2.12.0 (apache/parquet-format CHANGES.md, as of 2026-07-02). The format continues to evolve — a Variant type for semi-structured data was announced in February 2026 (parquet.apache.org blog, as of 2026-07-02).

Data page versions

Pages come in several types. A DictionaryPage stores the distinct values of a dictionary-encoded column. A DataPage carries the actual values. There are two versions of the data page. The original DataPage (V1) stores repetition levels, definition levels, and encoded values together in one compressed block. DataPageV2 separates the repetition and definition levels from the values, allows those levels to remain uncompressed while the values are compressed, and gives per-page control over whether compression is applied. This separation is what unlocks the delta encoding family described in the next section. An IndexPage type is reserved in the specification. The figure below contrasts the two data page versions.

Repetition and definition levels are the mechanism by which Parquet represents nested and repeated fields in a flat columnar store. This technique originates in the Dremel system described by Melnik and colleagues (VLDB 2010), the foundational paper for Parquet’s nested encoding. A definition level records how many of a column’s optional or nested ancestors are actually present for a given value, which is how nulls at any nesting depth are represented without storing placeholder values. A repetition level records at which nesting level a repeated field begins a new list. Together the two levels let a reader reconstruct arbitrarily nested records from flat columns.

Encoding and compression

Two distinct byte-reduction layers operate inside a Parquet column chunk, and conflating them is a common source of confusion. Encoding is a structural, format-aware transformation applied to the values of one column. Compression is a byte-level, general-purpose transformation applied to the already-encoded bytes of a page. The two stack: values are first encoded, then the encoded output is compressed. A reader reverses the order, decompressing a page and then decoding its values.

The encoding families

PLAIN encoding is the baseline. It writes values in their natural fixed-width or length-prefixed form with no attempt at reduction, and it exists as a fallback when no structural pattern can be exploited.

Dictionary encoding is the workhorse for columns with a limited number of distinct values, such as country codes, status strings, or category labels. The writer builds a dictionary of distinct values, stored in a DictionaryPage, and replaces each value in the data pages with a small integer index into that dictionary. Those indices are then written with the RLE_DICTIONARY encoding, which has enum value 8 in the specification, is valid for all physical types, and is the default for string columns (apache/parquet-format Encodings.md, as of 2026-07-02). If the dictionary grows beyond the configured dictionary page size — the default dictionary page size is 1 MB (parquet.apache.org configurations, as of 2026-07-02) — the writer abandons the dictionary for that column chunk and falls back to PLAIN. One dictionary page is written per column per row group.

The RLE / bit-packing hybrid encoding underlies dictionary indices and also carries the repetition and definition levels. It combines run-length encoding, which represents a repeated value as a value-plus-count pair, with bit-packing, which stores small integers using only as many bits as their range requires rather than a full 32 or 64 bits. The encoder switches between the two modes according to which is smaller for each run.

The delta encoding family targets numeric and byte-array columns and is a DataPageV2 feature, not used for V1 pages (parquet.apache.org encodings, as of 2026-07-02). DELTA_BINARY_PACKED, enum value 5, applies to INT32 and INT64 columns and stores the differences between consecutive values rather than the values themselves (apache/parquet-format Encodings.md, as of 2026-07-02). For a sorted or monotonic column such as a timestamp or an auto-incrementing identifier, those differences are small and highly repetitive, so the encoded output is far smaller than the raw integers. DELTA_BYTE_ARRAY, enum value 7, applies to BYTE_ARRAY and FIXED_LEN_BYTE_ARRAY columns; it encodes the lengths of the byte arrays using DELTA_BINARY_PACKED and then concatenates the bytes, and it is preferred over PLAIN for byte arrays (parquet.apache.org encodings, as of 2026-07-02). A related variant, DELTA_LENGTH_BYTE_ARRAY, separates the length stream from the data stream for strings without shared prefixes.

BYTE_STREAM_SPLIT, enum value 9, applies to INT32, INT64, FLOAT, DOUBLE, and FIXED_LEN_BYTE_ARRAY columns (apache/parquet-format Encodings.md, as of 2026-07-02). It does not reduce size by itself. Instead it scatters the K bytes of each value into K separate streams, placing all the first bytes together, then all the second bytes, and so on. Floating-point values that cluster around a common magnitude share high-order bytes, and grouping those bytes together gives the downstream compressor much more redundancy to exploit (parquet.apache.org encodings, as of 2026-07-02). This is a clear illustration of the encoding-then-compression stack: the encoding rearranges bytes so that the compression layer performs better.

The compression codecs

After encoding, each page is compressed with a block-level, general-purpose codec. The supported codecs are UNCOMPRESSED, SNAPPY, GZIP, LZ4 and LZ4_RAW, ZSTD, and BROTLI. Because each page is compressed independently, a reader can decompress exactly the pages a query touches and no others. The choice among codecs is a trade-off between compression ratio and processing speed, and the appropriate choice is workload-dependent rather than fixed.

Metadata, statistics, and pushdown

The performance of an analytical scan depends less on how fast the reader processes data than on how much data it manages to avoid reading at all. Parquet supports this avoidance through metadata stored in the footer and in optional index structures. Two mechanisms operate together: projection pushdown selects which columns to read, and predicate pushdown selects which rows — more precisely, which row groups and pages — to read.

Projection pushdown was described earlier. Because each column is stored as an independent column chunk, a reader consults the footer to find the byte ranges of the requested columns and reads only those ranges. No special index is required; the columnar layout makes it free.

Predicate pushdown is more involved and operates in three pruning tiers, each addressing a case the previous tier cannot handle. The figure below shows the tiers as a funnel that progressively narrows the data a query must read.

The three pruning tiers

The first tier is row-group statistics. The footer stores, for each column chunk, the minimum value, the maximum value, and the null count. When a query filters on a column, the reader compares the predicate against these statistics. If a row group’s minimum and maximum for the filtered column cannot possibly satisfy the predicate, the reader skips the entire row group — up to 128 MB of data — without reading any of its pages. This tier is coarse but very cheap, since the statistics are already in the footer the reader parsed on open.

The second tier is the Page Index, which brings the same idea down to page granularity. The Page Index was introduced in Parquet 1.11 and is used by Spark 3.2.0 and later (CERN Databases blog and parquet-format PageIndex.md, as of 2026-07-02). It consists of two structures. The ColumnIndex stores the minimum and maximum values for each page in a column chunk, letting the reader identify exactly which pages could contain matching rows. The OffsetIndex maps a row index to the byte offset of the page that contains it, so once the ColumnIndex has selected the relevant pages, the reader seeks directly to them. Together these let a reader skip individual pages within a column chunk that survived the row-group filter.

The third tier is the Bloom filter, which handles a case the range-based tiers cannot. Minimum and maximum statistics prune well when the filtered values fall outside a range, but they are useless for equality predicates on high-cardinality columns whose values are scattered across the whole domain. A filter such as user_id = 90422 will match the min/max range of almost every row group even though only a handful of rows qualify. A Bloom filter is a compact probabilistic structure that answers set membership. Parquet supports exactly one representation, the Split Block Bloom Filter (SBBF), whose block size is 256 bits, arranged as eight contiguous 32-bit words (apache/parquet-format BloomFilter.md, as of 2026-07-02). It answers a membership query with either “definitely not present” or “probably present” — it produces no false negatives, only occasional false positives (apache/parquet-format BloomFilter.md, as of 2026-07-02). When the answer is “definitely not present,” the reader skips the row group. Dictionary pages can serve a similar membership-pruning role, since a value absent from a column chunk’s dictionary cannot appear in that chunk.

Apache Arrow and the Parquet-to-Arrow bridge

Parquet describes data at rest. Once a reader decodes a Parquet page, the values must live somewhere in memory in a form suitable for computation. That form is Apache Arrow. Arrow defines a language-independent, in-memory columnar layout for both flat and nested data, designed for efficient analytics on CPUs and GPUs (arrow.apache.org Columnar.html, as of 2026-07-02). The latest Apache Arrow release is 24.0.0, published on 2026-04-21, comprising 259 resolved issues and 325 commits from 57 contributors over roughly three months (arrow.apache.org blog, as of 2026-07-02).

Where Parquet optimizes for size on disk, Arrow optimizes for access speed in memory. An Arrow array stores a column’s values in a contiguous buffer with a separate validity bitmap indicating which entries are null. This layout is predictable and aligned, so a processor can iterate over it sequentially and apply vectorized instructions that operate on many values per cycle. The Parquet-to-Arrow bridge decodes Parquet pages directly into Arrow arrays: a library such as pyarrow reads a Parquet file and produces Arrow Tables, translating the on-disk encodings into the in-memory layout in one step.

Zero-copy and the C Data Interface

The most consequential property of Arrow is that independent libraries can agree on the exact byte layout of a column and therefore share it without copying. When a query engine, a dataframe library, and a machine-learning framework all represent a column as the same Arrow buffer, one can hand the column to another by passing a pointer rather than serializing, transmitting, and deserializing the data. This is the mechanism the Arrow C Data Interface formalizes. It enables zero-copy sharing between runtimes within a single process through memory pointers, and its only dependency is a stable C application binary interface (ABI) — a fixed set of C structures that every participant understands (arrow.apache.org CDataInterface.html, as of 2026-07-02).

This capability is why an in-process analytical stack can combine tools that were developed independently without incurring a translation tax between them. A query engine can execute a filter, hand the resulting Arrow columns to a dataframe library for a transformation, and pass those to a machine-learning framework, all without any of the three re-serializing the data. The comparison of in-process engines that read Parquet and exchange Arrow buffers is examined in the companion post on DuckDB versus Polars for in-process analytics.

Three ways Arrow data moves

Arrow data is exchanged through several mechanisms, and distinguishing them prevents confusion. The C Data Interface moves data within a single process by passing pointers to buffers already in memory. Arrow IPC, also known as the Feather file format, serializes Arrow data into a byte stream for transmission across a network or for writing to a file — it is a wire and storage format for Arrow itself. Parquet, finally, is the durable at-rest format optimized for long-term storage rather than for immediate computation. Each serves a different boundary: within a process, across a process or network, and at rest. The figure below maps the three mechanisms to the boundaries they cross.

Table formats such as Apache Iceberg, Delta Lake, and Apache Hudi layer a metadata and transaction management system on top of Parquet data files rather than replacing them; the distinction between the file layer and the table layer is examined in the comparison of Iceberg, Delta Lake, and Hudi table formats. A transformation tool that materializes analytical models frequently writes those models as Parquet, a pattern covered in the guide to dbt transformation pipelines.

Practical guidance for tuning

The internals described above translate into a small set of tuning decisions. None of them has a single correct answer; each depends on the storage medium, the reader’s memory budget, and the query patterns.

Row-group sizing

The default row-group size is 128 MB (parquet.apache.org configurations, as of 2026-07-02). A larger row group amortizes footer and metadata overhead across more rows and produces longer contiguous reads, which suits object storage where each request has latency. A smaller row group increases the granularity of row-group statistics pruning and reduces the memory a reader must hold to process one row group. When files are read from object storage, aligning the row-group size with the object store’s block or multipart size reduces the number of range requests. When a reader has a constrained memory budget, a smaller row group prevents it from having to buffer an entire 128 MB band per column.

Sorting to help encoding and pruning

Sorting the data by a frequently filtered column improves two mechanisms at once. First, a sorted numeric column has small, repetitive deltas, which makes DELTA_BINARY_PACKED encoding highly effective. Second, sorting tightens the minimum and maximum ranges within each row group and page, so that row-group statistics and the Page Index prune far more aggressively — an unsorted column tends to span the full value range in every row group, defeating range-based pruning. Sorting therefore benefits both file size and scan speed.

When to enable Bloom filters

A Bloom filter earns its space only on a high-cardinality column that is queried with equality predicates, such as a user identifier, a session token, or an email address. On such a column, minimum and maximum ranges overlap across every row group and cannot prune, so the Split Block Bloom Filter provides the only membership-based pruning available. On a low-cardinality column, a Bloom filter is wasteful, because dictionary encoding and range statistics already prune those columns effectively. Bloom filters should be enabled selectively, per column, based on query patterns.

Choosing a codec and page version

For hot data queried frequently, Snappy’s fast decompression favors query latency. For cold or archival data, Gzip’s higher ratio reduces storage cost at the expense of slower reads. Zstd offers a tunable middle ground and is increasingly chosen as a default because its ratio approaches Gzip’s at speeds closer to Snappy’s; the exact ratio remains workload-dependent and should be measured on the real dataset. DataPageV2 is required to use the delta encoding family, so writers that store sorted integer or timestamp columns benefit from enabling V2 pages. These file-level choices are typically set once in the writer configuration of the job that produces the data, whether that job runs under an orchestrator such as the one described in the guide to Apache Airflow pipeline orchestration or as part of an ingestion pipeline like the one in the walkthrough of an InfluxDB-to-Iceberg data pipeline.

Frequently Asked Questions

What is the difference between Apache Parquet and Apache Arrow?

Parquet is a format for storing columnar data on disk or in object storage, optimized for space efficiency through encoding and compression. Arrow is a format for holding columnar data in memory, optimized for fast processing on CPUs and GPUs. A reader decodes Parquet files into Arrow arrays to compute on them. The two are designed together: Parquet handles data at rest and Arrow handles data in memory.

What is the default row-group and page size in Parquet?

The default row-group size is 128 MB, set by the parquet.block.size configuration at 134217728 bytes, and the default page size is 1 MB, set by parquet.page.size at 1048576 bytes (parquet.apache.org configurations, as of 2026-07-02). A row group is a horizontal band of rows processed independently, and a page is the smallest unit that must be read and decompressed in full to access any record inside it.

How does predicate pushdown skip data in a Parquet file?

Predicate pushdown prunes data in three tiers. Row-group statistics in the footer store per-column min, max, and null count, letting a reader skip an entire row group. The Page Index refines this to individual pages using per-page min and max values with a row-index-to-offset map. The Split Block Bloom Filter handles equality on high-cardinality columns where ranges cannot prune, answering “definitely not present” or “probably present” (apache/parquet-format BloomFilter.md, as of 2026-07-02).

Why do Parquet files need both encoding and compression?

Encoding and compression are two separate layers that stack. Encoding is a structural, format-aware transformation of one column’s values, such as dictionary encoding or delta encoding, that exploits the column’s type and distribution. Compression is a general-purpose byte-level codec, such as Snappy or Zstd, applied to the already-encoded page. Encoding often rearranges bytes so the compressor finds more redundancy; BYTE_STREAM_SPLIT is a clear example, adding no size reduction itself but improving downstream compression (parquet.apache.org encodings, as of 2026-07-02).

What does the Arrow C Data Interface do?

The C Data Interface lets separate libraries in the same process share Arrow column buffers without copying, by passing pointers through a small set of standard C structures. Its only requirement is a stable C application binary interface (arrow.apache.org CDataInterface.html, as of 2026-07-02). This lets a query engine, a dataframe library, and a machine-learning framework exchange columns without serializing and deserializing the data at each handoff.

References

• Apache Parquet format specification and parquet.thrift — apache/parquet-format, spec version 2.12.0 (as of 2026-07-02)
• Parquet encodings reference — parquet.apache.org documentation (as of 2026-07-02)
• Parquet Split Block Bloom Filter specification — apache/parquet-format BloomFilter.md (as of 2026-07-02)
• Apache Arrow columnar format specification — arrow.apache.org (v24.0.0, as of 2026-07-02)
• Apache Arrow C Data Interface — arrow.apache.org (v24.0.0, as of 2026-07-02)
• Dremel: Interactive Analysis of Web-Scale Datasets — Melnik et al., VLDB 2010, the foundational source for Parquet’s repetition/definition-level nested encoding

Conclusion

A Parquet file is not an opaque blob but a carefully layered structure whose every level serves a purpose. The file-to-row-group-to-column-chunk-to-page hierarchy makes projection pushdown free and gives predicate pushdown its units of pruning. The encoding layer exploits the type and distribution of each column, and the separate compression layer removes the byte-level redundancy that remains. The footer’s statistics, the Page Index, and the Split Block Bloom Filter combine into a three-tier funnel that lets a reader avoid the majority of the data a query does not need. Apache Arrow completes the picture by defining the in-memory layout into which those pages decode, and its C Data Interface lets independent libraries share columns without paying a serialization cost.

These internals are not academic. Row-group sizing, sorting by a filtered column, selective Bloom filters, DataPageV2 for delta encodings, and codec choice are all decisions that follow directly from understanding the mechanism. An engineer who knows what the bytes look like can reason about why a scan is slow and what to change, rather than treating the format as a black box. The formats continue to evolve — Parquet at specification version 2.12.0 with a new Variant type announced in February 2026, and Arrow at release 24.0.0 from April 2026 (as of 2026-07-02) — but the columnar principles described here remain the stable foundation beneath every analytical query that touches them.

On a single 10-million-row Parquet file of about 121 MiB, five analytical queries ran between 1.85 and 5.14 times faster under DuckDB than under Polars, and DuckDB’s peak memory stayed under 130 MiB while Polars reached roughly 1.1 GiB on the heaviest query. Those figures were measured for this comparison on an Apple M2 Pro laptop and are reported in full, with the raw terminal output, further down. They describe one modest, in-memory, single-file workload rather than a universal ranking, and the same two engines trade the lead at larger scales in independent public benchmarks. That tension is the reason this comparison pairs first-hand measurement with the published evidence: DuckDB and Polars are close enough that the right choice depends on scale, workload, memory behavior, and the interface a team prefers, not on a single headline number.

DuckDB and Polars have become two of the most widely adopted tools for analytical work that runs inside a single application process, rather than against a separate database server or a distributed cluster. Both read columnar files such as Parquet directly, both execute queries over an in-memory layout derived from the Apache Arrow format, and both are fast enough that many data teams now reach for one of them before considering a cluster engine such as Apache Spark. They are, however, built around different ideas: DuckDB is a relational database that speaks SQL, while Polars is a DataFrame engine driven by an expression API and written in Rust. This comparison examines their execution models, the memory behavior each shows under load, a reproducible benchmark measured on one machine, the public benchmark evidence at larger scale, and the often-overlooked fact that the two can hand data to each other with no copy.

Two engines, one process: what “in-process analytics” means

The term in-process describes software that runs within the address space of the calling application rather than as a separate service that the application contacts over a network or a socket. A traditional analytical database such as a data warehouse runs as a server: an application opens a connection, sends SQL over the wire, and receives rows back. An in-process engine instead loads as a library inside the program — a Python interpreter, for example — and operates on data in the same memory the program already holds. There is no server to start, no port to manage, and no connection pool to tune.

DuckDB describes itself as “an in-process analytical (OLAP) database.” OLAP, or online analytical processing, refers to read-heavy workloads that scan and aggregate large numbers of rows — the opposite of online transaction processing (OLTP), which favors many small, indexed reads and writes. Polars describes itself as “an analytical query engine written for DataFrames.” A DataFrame is a two-dimensional table abstraction, familiar from libraries such as pandas, in which columns carry typed values and operations are expressed as transformations of whole columns. The two descriptions point at the same workload — analytical queries over tabular data — approached from a relational direction (DuckDB) and a DataFrame direction (Polars). (Sources: duckdb.org/why_duckdb; pypi.org/project/polars, as of 2026-06-22.)

The distinction from a server architecture matters for how a system is built and operated. The figure below contrasts the two deployment shapes.

Neither engine is a distributed system, and this is the single most useful framing for someone deciding when to use them. Apache Spark, the dominant cluster engine for large-scale processing, distributes work across many machines coordinated by a driver and scheduler. That model is appropriate when a dataset genuinely does not fit on one machine or when a workload must be elastic across a fleet. DuckDB and Polars instead exploit a single machine well — every core, the full memory hierarchy, and fast local or cloud storage — and increasingly that is sufficient. Cloud instances now offer hundreds of gigabytes of RAM, and many analytical datasets are smaller than teams assume. When a job fits comfortably on one node, an in-process engine removes the operational weight of a cluster while often finishing faster, because no time is lost to network shuffles or task scheduling.

Because both engines sit at the compute layer, they read from the storage and table-format layer below them rather than replacing it. A team standardizing on open table formats may want to review how those formats differ in the comparison of Apache Iceberg, Delta Lake, and Hudi table formats, since DuckDB and Polars are the engines that would read tables stored in those formats.

Architecture compared: SQL versus DataFrame, eager versus lazy

The two engines share a foundation and diverge above it. Both are columnar, meaning they store and process data column by column rather than row by row. Column-oriented layout suits analytical queries because such queries typically touch a few columns across many rows; reading only the needed columns reduces input/output and lets the processor work on long, uniform runs of values. Both are also vectorized: instead of evaluating one value at a time, they process a batch of values — a vector — in each step of execution, which keeps the processor’s pipeline full and amortizes the overhead of interpreting each operation. DuckDB states that in its engine “queries are still interpreted, but a large batch of values (a ‘vector’) are processed in one operation.” Polars lists “SIMD” operations, referring to Single Instruction, Multiple Data, a processor capability that applies one instruction to several data elements at once. (Sources: duckdb.org/why_duckdb; pypi.org/project/polars, as of 2026-06-22.)

SQL engine versus DataFrame engine

The most visible difference is the interface. DuckDB executes SQL, the declarative query language used by relational databases for five decades. A user writes what result is wanted and the engine decides how to produce it. DuckDB also provides “ACID guarantees through our custom, bulk-optimized Multi-Version Concurrency Control (MVCC)” — ACID being the set of transaction properties (atomicity, consistency, isolation, durability) that keep concurrent changes correct, and MVCC being a technique that lets readers and writers proceed without blocking each other by keeping multiple versions of a row. (Source: duckdb.org/why_duckdb, as of 2026-06-22.)

Polars exposes “a DataFrame expression API plus a SQL interface.” Its primary interface is the expression API, in which transformations are composed as method calls and column expressions in the host language. Polars is “written in Rust,” uses the “Apache Arrow Columnar Format,” and offers frontends in “Python, Rust, NodeJS, R, [and] SQL.” A DataFrame engine appeals to programmers who prefer composing transformations as code, with the full control flow and tooling of a general-purpose language, over embedding SQL strings. (Source: pypi.org/project/polars, as of 2026-06-22.)

Parallelism is realized differently. DuckDB uses “morsel-driven parallelism,” a scheduling approach in which input data is divided into small, fixed-size chunks called morsels that worker threads pull and process, so that load balances across cores without a rigid up-front partitioning. Polars lists “multi-threaded” execution combined with SIMD. Both aim at the same outcome — saturating all available cores on one machine — through different scheduling strategies. (Sources: duckdb.org/why_duckdb; pypi.org/project/polars, as of 2026-06-22.)

Eager versus lazy evaluation

Polars makes an explicit distinction that shapes how performant code is written: eager versus lazy evaluation. In eager mode, each operation runs immediately and returns a materialized result, which is convenient for interactive exploration. In lazy mode, operations build a query plan that is not executed until a terminal call — collect() — is reached. Deferring execution lets the optimizer rewrite the whole plan before any work happens: it can push filters down to the data source so that fewer rows are read, prune unused columns, and reorder operations. Polars documents both modes as “Lazy | Eager execution” with “Query optimization.” The measured benchmark below uses the lazy API for Polars for exactly this reason. (Source: pypi.org/project/polars, as of 2026-06-22.)

DuckDB does not expose an eager/lazy switch because SQL is declarative by nature: a complete statement is handed to the optimizer, which always sees the whole query before execution. The lazy mode of Polars is, in effect, a way to recover that whole-query view inside an imperative DataFrame API. For a data engineer, the practical guidance is that Polars code intended for production should use the lazy API, since eager chains forgo the optimizations that make the engine fast on large inputs.

Streaming and larger-than-memory execution

A defining concern for single-node engines is what happens when a dataset is larger than available RAM. Polars supports processing data “in a streaming fashion,” enabled with collect(engine='streaming'), which executes the query in chunks so that the full dataset need not be resident at once. DuckDB, as a database, has long been designed to spill intermediate state to disk when memory is exhausted, allowing queries to complete on data larger than memory. Both engines therefore offer a path beyond the RAM ceiling, though, as the benchmark section shows, their default memory footprints can differ substantially. The measured benchmark below deliberately uses a dataset that fits in RAM, so it exercises the in-memory path rather than streaming or spilling. (Sources: pypi.org/project/polars; duckdb.org/why_duckdb, as of 2026-06-22.)

Both engines build on the Apache Arrow columnar memory format, an open standard for representing tabular data in memory so that different tools can share buffers without reformatting. Arrow is the reason the two engines interoperate so cheaply, a point developed in the interoperability section. The same Parquet files — Parquet being the dominant open columnar file format on disk — can be read directly by either engine, which means a team is not locked into a proprietary storage format by either choice.

A measured benchmark on one machine

Public benchmarks are useful but abstract; a reproducible measurement on a known machine gives a concrete anchor. The benchmark described in this section was run on a single laptop specifically for this comparison, with every recorded number taken directly from the program’s output. The complete code, the dataset generator, the environment capture, and the raw terminal log are published so the run can be repeated and audited (see the References section). The purpose is not to declare a winner but to show, honestly, how the two engines behaved on one well-defined in-memory workload.

Test environment

All timings were collected on the hardware and software listed below, captured programmatically with sw_vers, sysctl, and the package manager’s list command. Both engines ran with their default thread settings, which resolved to 10 threads on this machine.

Benchmark setup

The dataset was generated by a seeded script (seed 20260719) and written to a single Snappy-compressed Parquet file. Its shape was recorded at generation time:

• Rows: 10,000,000
• File size: 126,976,888 bytes (121.09 MiB)
• Columns: ts (microsecond timestamp over a 90-day window, unsorted), user_id (32-bit integer, Zipf-skewed across 500,000 distinct users), category (one of 10 strings, dictionary-encoded, drawn from a skewed distribution), amount (64-bit float, log-normal, rounded to cents), and flag (boolean, roughly 30 percent true).
• Dimension table: a small dims table of 10 rows (category, region, tier) for the join query.

Five queries were defined with identical intent for both engines. DuckDB ran SQL over read_parquet(...); Polars used scan_parquet(...) lazy frames collected with .collect(). Each query fully materialized its result, so the timing includes producing the complete output, not just planning it.

The measurement protocol was designed to keep the numbers clean. Each engine-query pair ran in a fresh Python subprocess, so the peak-memory reading for one query could not be inflated by memory another query had left resident. Each query ran one warmup pass followed by three timed passes, and the reported time is the median of the three timed passes, measured with a wall-clock timer around the full materialization call. Peak memory is the maximum resident set size (RSS) observed during the timed runs, sampled about every two milliseconds by a background thread and re-anchored at the start of each run. One caveat on the join query is worth stating in advance: in the Parquet file, category is dictionary-encoded and Polars reads it as a categorical type, whereas the dimension table’s key is a plain string, so both keys were cast to string before the join. That cast cost is included in the Polars timing for q4 and is discussed with the results.

Query time results

On this dataset DuckDB was faster on every query. The ratio in the final column is the Polars median divided by the DuckDB median, so a value of 3.05 means DuckDB completed that query about three times faster.

The gap ranged from a factor of 1.85 on the simple full-scan count to a factor of 5.14 on the join. The join result deserves the caveat noted earlier: on the Polars side the join key was cast from a categorical type to a string before the operation, and that cast is counted in the 0.1763-second figure. Part of the 5.14 ratio therefore reflects a data-typing detail rather than pure engine speed, and a schema that avoided the cast would likely narrow it. The smallest gap, on scan_count_distinct, is consistent with a query dominated by reading and hashing a single high-cardinality column, where both engines spend most of their time in similar work.

Peak memory results

The memory picture was more lopsided than the timing picture. DuckDB’s peak resident memory stayed between 77 and 123 MiB across the five queries, close to the on-disk size of the data. Polars, running its default in-memory collection, ranged from 182 MiB to 1,086 MiB, with the two grouping-heavy queries — time_bucket and join_dims — the most memory-intensive.

The raw terminal output below is copied verbatim from the run log, so the numbers in the tables and charts can be checked against their source.

==============================================================================
RESULTS  (median of 3 timed runs)
==============================================================================
query                 DuckDB med(s)  Polars med(s)   ratio P/D
--------------------------------------------------------------
filtered_groupby             0.0168         0.0511        3.05
highcard_topn                0.0285         0.0940        3.30
time_bucket                  0.0396         0.1491        3.76
join_dims                    0.0343         0.1763        5.14
scan_count_distinct          0.0210         0.0388        1.85

PEAK RSS  (max over timed runs, MiB)
query                    DuckDB MiB     Polars MiB
--------------------------------------------------
filtered_groupby               77.0          354.5
highcard_topn                 123.2          463.6
time_bucket                   120.9         1086.1
join_dims                      77.1          886.9
scan_count_distinct          107.0          181.8

ARROW INTEROP  (duckdb.arrow -> polars.from_arrow, median of 3)
  rows_transferred     = 3,001,101
  duckdb -> arrow  (s) = 0.0018
  arrow  -> polars (s) = 0.2055
  round trip total (s) = 0.2073

What the measurements show — and what they do not

Read within its boundaries, this run gives a clear result: for a 10-million-row file that fits in RAM, default DuckDB was faster on all five queries and used far less memory than default Polars. The memory finding is the more decisive of the two, since a query that peaks near 1 GiB rather than near 120 MiB changes how many concurrent jobs a machine can host, even when both finish in a fraction of a second.

The result also corrects an impression that a documentation-only reading of the two engines can leave. Because the Polars project’s own published benchmark shows its streaming engine winning at a 10 GB scale factor, it is tempting to conclude that Polars is the faster choice at small scale in general. This measured workload is smaller still — about 0.12 GB — and default in-memory Polars was slower on every query. The lesson is that a benchmark result belongs to a specific configuration: the Polars streaming figure reflects a particular engine mode and query suite, not a universal small-scale advantage, and the default lazy-then-collect path measured here behaves differently. Any claim that one engine is simply “faster at small data” should be treated with suspicion until the configuration is named.

Several limits keep this from being a verdict. The dataset is small and fits entirely in memory, so it never exercises streaming or disk spilling, where the balance can shift. Polars ran in its default collection mode rather than engine='streaming', and its higher memory reflects that default; a streaming collect would trade some speed for a lower footprint. The join ratio is partly a data-type artifact, as noted. The sample is three timed runs on one laptop with warm file-system cache. And a single-file layout favors neither engine’s partition handling. The independent public benchmarks in the next section, run at 10 GB and 100 GB, show the two engines much closer and the lead changing hands, which is the necessary counterweight to a small in-memory test.

Larger-scale context: PDS-H and a memory stress test

Two public benchmarks extend the picture to scales a laptop test cannot reach, and both must be read with their conditions in mind. Neither is a vendor-neutral, audited TPC-H comparison of the two engines, because none existed as of writing, so they are context rather than a final ranking.

The PDS-H throughput benchmark

The Polars project publishes a benchmark based on PDS-H, a workload derived from TPC-H. TPC-H is a long-standing industry decision-support benchmark consisting of analytical queries over a synthetic dataset whose size is set by a “scale factor” (SF); SF-10 is roughly ten gigabytes of raw data and SF-100 roughly one hundred. The PDS-H total execution times reported by the Polars project (May 2025) are shown below.

Two patterns stand out. First, both DuckDB and Polars are an order of magnitude faster than pandas, PySpark, and Dask across these scales, which supports the broader claim that an in-process columnar engine outperforms both a single-threaded DataFrame library and a cluster engine on a single node at these sizes. Second, the lead changes with scale: Polars streaming was fastest at SF-10 (3.89 s versus DuckDB’s 5.87 s), while DuckDB was fastest at SF-100 (19.65 s versus Polars streaming’s 23.94 s). The collapse of Polars in-memory from 9.68 s at SF-10 to 152.27 s at SF-100 also shows why the streaming engine matters as data grows — and why the in-memory mode measured on the small laptop file is not the mode to use at scale. (Source: pola.rs/posts/benchmarks, as of 2026-06-22.)

The Parquet memory stress test

Peak memory determines whether a job fits on a given machine at all. An independent benchmark published on the codecentric blog (published 2026-01-20, updated 2026-02-02) scaled a single Parquet file from roughly 2 GB to 140 GB and measured both execution time and peak memory. On execution time the two engines were very similar across the range, with DuckDB roughly one second faster at the largest scale. The revealing result was peak memory at 140 GB: DuckDB held roughly 1.3 GB by default while default Polars used roughly 17 GB on the same file, yet forcing asynchronous reads brought Polars down to roughly 750 MB, below DuckDB. Separately, partitioning the 140 GB dataset into 72 smaller files cut DuckDB’s memory by roughly eight times and default Polars’ by roughly four times, showing that file layout strongly influences memory regardless of engine. (Source: codecentric.de blog, as of 2026-06-22.)

The stress test and the laptop measurement agree on direction: DuckDB is conservative with memory out of the box, and default Polars is not. They differ on whether that gap is intrinsic — the stress test shows Polars can be tuned below DuckDB with asynchronous reads at 140 GB, a lever the small in-memory test did not pull. Together they support the same discipline: measure under conditions that resemble the real workload, since the same “measure under your own conditions” principle applies when selecting data stores more broadly, as the comparison of databases for preprocessed time-series data walks through for a related class of workload.

Interoperability: zero-copy handoff over Apache Arrow

A point often missing from “DuckDB versus Polars” discussions is that the two are not mutually exclusive within one program. Because both represent data in the Apache Arrow columnar format, they can share the same in-memory buffers with little or no copying. The official DuckDB documentation states that “DuckDB can read Polars DataFrames and convert query results to Polars DataFrames. It does this internally using the efficient Apache Arrow integration.” In practice, a Polars DataFrame held in a variable can be referenced directly by name inside a SQL query — duckdb.sql("SELECT * FROM df") — and the result can be converted back to a Polars DataFrame with .pl(), or to a Polars LazyFrame with .pl(lazy=True). The pyarrow package is required for this path. (Source: duckdb.org/docs/current/guides/python/polars.html, as of 2026-06-22.)

Zero-copy means the handoff transfers ownership of, or a reference to, the existing memory buffer rather than serializing and reallocating the data. The figure below shows a single Arrow buffer being read by both a Polars DataFrame and a DuckDB query.

The benchmark measured this handoff as well. A filtered result of 3,001,101 rows was moved from DuckDB into an Arrow table and then into a Polars DataFrame. Exporting from DuckDB to Arrow took a median of 0.0018 seconds — effectively free, consistent with a reference handoff rather than a copy. Reconstructing a Polars DataFrame from that Arrow table (polars.from_arrow) took a median of 0.2055 seconds, so the full round trip was 0.2073 seconds for just over three million rows. The practical reading is that the boundary between the two engines is cheap relative to re-reading the data or serializing it across a process, but “zero-copy” does not mean literally zero time; the Polars ingestion step still does measurable work. Even so, moving three million rows in about a fifth of a second is inexpensive next to the cost of the analytical queries themselves at larger scales.

This interoperability changes the decision calculus. A pipeline can perform DataFrame-style feature preparation in Polars, hand the frame to DuckDB for a complex multi-table SQL join and aggregation, and receive the result back as a Polars frame for the next step — all within one process and without paying a full serialization cost at each boundary. The two engines become complementary stages rather than competing choices.

Both engines also fit cleanly into the wider data ecosystem. They read Parquet from local disk and from object storage such as Amazon S3, which is the common substrate for analytical data lakes. DuckDB has an adapter, dbt-duckdb, that lets it serve as the execution engine for transformation models; teams already using that framework can read about the model-based workflow in the guide to building transformation pipelines with dbt. An in-process engine is also a natural transform step inside a scheduled workflow, where an orchestrator triggers the job; the patterns for that are covered in the guide to orchestrating data pipelines with Apache Airflow. When the analytical layer is fed by streaming ingestion, the upstream side is described in the guide to change data capture with Debezium and Kafka.

Choosing: when DuckDB, when Polars, when both

Because the engines trade the performance lead by scale and workload, the most reliable basis for choosing is fit — to the team’s skills, the surrounding system, and the shape of the work — rather than a single benchmark number. The measured laptop test is a reminder that on small in-memory data DuckDB is fast and frugal by default, while the public benchmarks are a reminder that the picture narrows and can reverse at scale. The decision flow below summarizes the practical signals.

The table below condenses the same guidance into a side-by-side reference.

A reasonable default is to choose by interface and team skills first: a team comfortable in SQL, or one that wants transactional guarantees and conservative memory out of the box, will find DuckDB the lower-friction option, while a team writing DataFrame-style transformation code, preparing features for machine learning, or working in Rust will find Polars more natural. When a single pipeline genuinely contains both shapes of work, the low-cost Arrow bridge makes “use both” a sound engineering choice rather than a compromise. The one case where neither is the right answer is data that cannot be made to fit on a single large machine even with streaming or spilling; that workload still belongs on a distributed engine.

Frequently Asked Questions

Which was faster in the measured benchmark, DuckDB or Polars?

On the 10-million-row, 121 MiB Parquet file used here, DuckDB was faster on all five queries — by 1.85 times on a full-scan count and up to 5.14 times on the join — and used far less peak memory (77 to 123 MiB, versus 182 to 1,086 MiB for default Polars). Two qualifications matter: the file fits entirely in RAM, so streaming and disk spilling were never exercised, and part of the largest ratio comes from a categorical-to-string cast on the Polars join key. At 10 GB and 100 GB in the Polars-published PDS-H benchmark, the two engines are much closer and the lead changes hands. Treat the laptop result as one honest data point about a small in-memory workload, not a general ranking. (Measured 2026-07-19; code and raw results linked in the References.)

Can DuckDB and Polars be used together?

Yes. Both represent data in the Apache Arrow columnar format, so they exchange data with little copying. A Polars DataFrame in scope can be queried directly by variable name in SQL, for example duckdb.sql("SELECT * FROM df"), and a DuckDB result converts back to a Polars DataFrame with .pl() or to a LazyFrame with .pl(lazy=True). In the benchmark here, moving a 3,001,101-row result from DuckDB into Arrow took about 0.0018 seconds, and the full round trip into a Polars DataFrame took about 0.2073 seconds. The pyarrow package is required. (Source: duckdb.org/docs/current/guides/python/polars.html, as of 2026-06-22.)

Does the benchmark mean DuckDB is the better choice overall?

No. It means DuckDB was faster and more memory-frugal on one small in-memory, single-file workload with Polars in its default collection mode. The result does not cover larger-than-memory data, streaming collection, partitioned files, or schemas that avoid the join-key cast, and independent large-scale benchmarks show the engines much closer. The durable basis for choosing is interface preference (SQL versus DataFrame), default memory behavior, transactional needs, and ecosystem fit, with the option to use both engines together over Arrow when a pipeline contains both styles of work.

References

• Benchmark code and raw results (dataset generator, benchmark orchestrator, environment capture, and full terminal log for the measured DuckDB vs Polars run reported here), measured 2026-07-19.
• DuckDB — Why DuckDB (in-process OLAP, columnar-vectorized execution, MVCC, morsel-driven parallelism), as of 2026-06-22.
• Polars — PyPI project page (analytical query engine for DataFrames, Apache Arrow format, lazy/eager execution, SIMD, streaming), as of 2026-06-22.
• Polars — PDS-H benchmark (TPC-H-derived, May 2025; results not comparable to certified TPC-H), as of 2026-06-22.
• DuckDB documentation — Integration with Polars (zero-copy Apache Arrow handoff, .pl() conversion), as of 2026-06-22.

Conclusion

DuckDB and Polars represent two routes to the same destination: fast analytical computation inside a single process, over open columnar data, on one machine. DuckDB approaches the problem as a relational database — SQL, ACID transactions through MVCC, conservative default memory, and morsel-driven parallelism — while Polars approaches it as a Rust DataFrame engine with an expressive expression API, explicit lazy evaluation, and SIMD-accelerated multi-threading. The benchmark measured here shows that on a small in-memory file DuckDB was faster on every query and much more frugal with memory in its default mode, while the independent public benchmarks show the two engines close and the lead changing hands at 10 GB and 100 GB. Both readings are true within their conditions, which is precisely why a single number should not decide the matter. Because both build on Apache Arrow, the most consequential and least discussed fact is that they interoperate at low cost, turning an apparent rivalry into a pairing. A sound selection therefore rests on interface preference, memory behavior, and ecosystem fit — and on measuring under conditions that resemble the intended workload — with the option, where a pipeline contains both relational and DataFrame work, to use both engines bridged over Arrow.

Open table formats turned commodity object storage into a transactional database layer, and the choice among the three principal implementations — Apache Iceberg, Delta Lake, and Apache Hudi — is one of the foundational decisions in a modern lakehouse. The decision is consequential because the table format governs how data is written, updated, queried, and shared across every engine that touches the storage layer. It is also a decision that has changed character over the past two years. Where engineers once treated the choice as a long-term lock-in, the three formats have begun to converge toward interoperability, so that the question is increasingly about defaults and operational fit rather than permanent commitment.

This article examines what a table format is and why it became necessary, how each of the three formats is designed internally, how they compare across the dimensions that matter in practice, and why the so-called format war is giving way to a model in which one physical dataset can be read as more than one format. It closes with concrete guidance on how to choose in 2026.

Why Table Formats Exist

A data lake in its simplest form is a collection of files — most commonly Apache Parquet, a columnar file format — stored in an object store such as Amazon S3, Google Cloud Storage, or Azure Blob Storage. A query engine reads those files and treats them collectively as a table. This arrangement is inexpensive and scalable, but the abstraction is weak. The storage layer knows only about files and directories; it has no concept of a table, a transaction, or a consistent snapshot. The problems that follow from this gap are what gave rise to table formats.

A table format is a specification and metadata layer that sits between a physical collection of data files and the query engines that read them, presenting that collection as a single logical table with database-like guarantees. The most important of those guarantees is ACID — atomicity, consistency, isolation, and durability — the set of properties that ensure a group of changes either fully applies or does not apply at all, that concurrent readers and writers do not observe partial state, and that committed data survives failures. Without a table format, the lake cannot offer these properties.

Consider what happens when a job appends a new day of data to a plain Parquet directory and fails halfway. Some files are written and some are not, and a reader that lists the directory during the failure observes an inconsistent table. The same fragility affects updates and deletes: changing a single record requires rewriting whole files, and there is no transactional boundary to make the change appear atomically. This is the “directory of files” problem — the table is defined by whatever files happen to be present when the engine lists the path, with no authoritative record of which files belong to a committed version.

Three further capabilities are absent from the plain-Parquet model and are central to why table formats were created. Schema evolution is the ability to add, rename, drop, or reorder columns without rewriting historical data or breaking existing readers. Time travel is the ability to query the table as it existed at a previous point in time, which supports reproducibility, auditing, and rollback. And efficient query planning depends on metadata that records which files contain which ranges of values, so the engine can skip files that cannot contain matching rows.

Table formats deliver these capabilities through a metadata layer — a set of files that record the table’s schema, its partitioning, and, critically, the exact list of data files that constitute each committed version of the table. A central concept in two of the three formats is the manifest: a metadata file that enumerates a group of data files together with statistics about their contents, such as the minimum and maximum value of each column within each file. With this information the engine performs file pruning, reading only the files that could contain rows matching a query predicate. The three formats examined below implement these ideas differently, and those differences drive their respective strengths.

Apache Iceberg: Hierarchical Metadata and Engine Neutrality

Apache Iceberg was created to provide a table format that no single engine owns and that scales to very large tables without the planning bottlenecks of earlier approaches. Its defining characteristic is a hierarchical metadata architecture and a specification-first design philosophy. The format is defined by a written specification, and any engine that implements the specification can read and write the tables, which is the basis for Iceberg’s wide adoption.

The metadata hierarchy

An Iceberg table is described by a tree of metadata files. At the root is a metadata.json file (often called the table metadata file) that records the current schema, partition specification, and a pointer to the current snapshot. Each snapshot references a manifest list, a file that enumerates the manifest files belonging to that snapshot. Each manifest file, in turn, tracks a set of individual data files and stores column-level statistics for them, such as per-column lower and upper bounds and null counts (Apache Iceberg documentation; Dremio, as of 2026).

This hierarchy has two practical consequences. First, query planning is efficient even for tables with millions of files, because the engine reads the manifest list and the relevant manifests rather than listing the object store. Second, because the statistics are recorded in the manifests, the engine can prune files during planning — eliminating files whose recorded value ranges cannot satisfy the query predicate — without opening the data files. This is the same principle that motivates careful storage selection for analytical workloads, a topic explored in the discussion of choosing databases for preprocessed time-series data.

Partition evolution and hidden partitioning

Two features distinguish Iceberg’s treatment of partitioning. Hidden partitioning means the table records how a column is transformed into partition values — for example, by truncating a timestamp to its day — so that queries filtering on the raw column automatically benefit from partition pruning without the writer or reader having to reference a separate partition column. Partition evolution means the partitioning scheme of a table can be changed over time without rewriting the historical data; new data is written under the new scheme while old data remains valid under the old one. This is significant operationally, because it removes one of the most expensive and disruptive migrations in traditional partitioned tables.

Engine reach

Iceberg’s specification-first, engine-agnostic design has produced the broadest multi-engine support of the three formats. As of 2026 it is read and written by Apache Spark, Apache Flink, Trino, and DuckDB, and is supported as a table format by managed warehouses including Snowflake and Google BigQuery (RisingWave; Dremio; Onehouse comparison guides, as of 2026). This breadth is the principal reason Iceberg has become the common denominator across the industry, a point developed in the convergence section. A practical example of landing data into Iceberg from an operational source is described in the guide to building an InfluxDB-to-Iceberg data pipeline with Telegraf, and stream processors such as Flink commonly write into Iceberg as part of complex event processing pipelines.

Delta Lake: A Transaction Log Born at Databricks

Delta Lake originated at Databricks and was designed first and foremost to bring transactional reliability to data stored for use with Apache Spark. Its integration with Spark is the deepest of the three formats, reflecting that shared origin (Databricks; Dremio, as of 2026). Where Iceberg organizes metadata as a tree of manifest files, Delta Lake organizes it as an ordered transaction log.

The transaction log

A Delta table keeps a directory named _delta_log alongside its data files. Each successful commit writes a new JSON file to this directory, numbered sequentially, recording the actions that the commit performed — which data files were added and which were removed, along with metadata changes. The current state of the table is obtained by replaying these JSON commits in order. To prevent the replay from growing unbounded, Delta Lake periodically writes a checkpoint in Parquet format that captures the full table state up to a given commit. A reader then loads the most recent checkpoint and applies only the JSON commits that follow it, which keeps state reconstruction efficient.

The transaction log model gives Delta Lake straightforward time travel — a reader can request the table state as of any commit version — and reliable concurrency control through optimistic commits against the log. Because the log is the single ordered authority on table state, the semantics map cleanly onto Spark’s execution model, which is part of why Delta Lake remains the most natural choice within Spark-centric and Databricks-centric environments. Transformation workflows that run on top of such tables, for instance with dbt-based transformation pipelines, treat the table as the consistent input and output of each model run.

UniForm and interoperability

For most of Delta Lake’s history its log-based metadata was readable only by Delta-aware engines, which limited reach relative to Iceberg. Delta UniForm addresses this directly. UniForm provides interoperability across Delta Lake, Iceberg, and Hudi by generating the metadata that other formats expect alongside the Delta log, and it supports the Iceberg REST catalog interface so that Iceberg clients can discover and read the tables (Databricks, as of 2026). In effect, a table written as Delta can be presented to an Iceberg reader without copying the underlying Parquet data. This is one of the developments that has softened the practical cost of choosing Delta in a mixed-engine environment, and it is examined further in the convergence section.

Apache Hudi: Built for Streaming Upserts

Apache Hudi was designed around a workload that the other two formats addressed only later: continuous ingestion of streaming data with frequent record-level updates. The name itself abbreviates “Hadoop Upserts Deletes and Incrementals,” which signals its orientation. As of 2026, Hudi offers the most mature tooling for pure streaming ingestion involving high-frequency upserts (Onehouse; lakeFS, as of 2026). An upsert is an operation that inserts a record if its key does not yet exist and updates the existing record if it does — the natural operation when a change-data-capture stream delivers a steady flow of inserts, updates, and deletes from an operational database. Feeds of this kind commonly arrive through change data capture with Debezium and Kafka or through a Kafka consumer that lands events into the lake.

Copy-on-Write and Merge-on-Read

Hudi offers two table types, and the choice between them is the central tuning decision when adopting the format. In a Copy-on-Write (CoW) table, an update rewrites the base file that contains the affected record, so every version of the data is materialized at write time. Reads are therefore fast and simple, because the engine reads finished base files, but writes are more expensive because they rewrite whole files. In a Merge-on-Read (MoR) table, an update is written to a delta log file associated with the base file rather than rewriting the base file immediately; readers merge the base file with its delta logs at query time. Writes are cheaper and lower-latency, which suits high-frequency upserts, at the cost of more work at read time until a background compaction merges the deltas into new base files.

Record-level indexing and table management

To apply an upsert, Hudi must locate the existing record for a given key quickly. It maintains record-level indexing for this purpose — a mapping from record keys to the files that contain them — so an incoming update can be routed to the correct file without scanning the whole table. This indexing is a meaningful part of why Hudi handles high-frequency update workloads efficiently. Hudi also includes built-in table management services: compaction merges MoR delta logs into base files, clustering reorganizes data to improve query locality, and cleaning removes obsolete file versions according to a retention policy. These services can run as part of the ingestion pipeline or as separate jobs, and orchestrating them alongside ingestion is a common use of a workflow scheduler such as Apache Airflow.

Hudi’s primary ingestion utility is its streaming ingestion tool, historically known as DeltaStreamer (now Hudi Streamer), which reads from sources such as Kafka and applies inserts, updates, and deletes to a Hudi table continuously. This tooling, together with record-level indexing and the MoR table type, is the basis for Hudi’s reputation in upsert-heavy streaming ingestion.

Native Iceberg output

Hudi has also moved toward interoperability. Tables created from Hudi version 0.14.0 onwards can be synced to Iceberg and/or Delta Lake through Apache XTable, and Hudi’s native Iceberg support lets a team use Hudi’s managed services — compaction, indexing, and Hudi Streamer ingestion — while outputting Iceberg-compatible tables for downstream consumers (Apache Hudi documentation, as of 2026). This means a pipeline can keep Hudi’s strengths on the write side while presenting Iceberg on the read side, a pattern that the convergence section places in context.

A Head-to-Head Comparison

The three formats overlap substantially in their core guarantees — all provide ACID transactions, schema evolution, and time travel — so the meaningful differences lie in their metadata models, their maturity for specific operations, and their ecosystem reach. The matrix below summarizes the breadth of engine support that distinguishes the formats, followed by a feature comparison and a workload mapping.

The grid is directional rather than a precise capability audit; engine support changes with each release, and warehouse vendors continue to add native read paths. The pattern it conveys is the one reported consistently across the comparison literature: Iceberg has the widest native reach, Delta Lake is strongest within Spark and reaches other engines primarily through UniForm, and Hudi is strong in Spark and Flink and reaches warehouses through translation. The detailed feature comparison follows.

The feature comparison clarifies that the formats are not interchangeable on every axis even though their core guarantees overlap. Partition evolution and hidden partitioning are genuine Iceberg differentiators; built-in table management is a genuine Hudi differentiator; and the deepest Spark integration remains a Delta Lake characteristic. The second table maps common workloads to the format that most naturally fits each, before interoperability is taken into account.

The Convergence Story

The most consequential change in this area is not a new capability in any single format but the erosion of the boundaries between them. Several developments, taken together, have moved the ecosystem from competition toward interoperability, and they explain why the choice of format is now less of a permanent commitment than it was.

The Databricks–Tabular acquisition

In a development that reframed the competitive landscape, Databricks acquired Tabular — the startup founded by the original creators of Apache Iceberg — for more than one billion dollars, a deal revealed on June 4, 2024 (Databricks blog; TechTarget, as of 2024-06-04). Because Databricks is the company behind Delta Lake, the acquisition brought the architects of the rival format inside the same organization and signaled an intent to support both formats rather than insist on one. It is widely read as the moment the “format war” framing began to lose force in favour of interoperability.

UniForm, Hudi output, and Apache XTable

Three technical mechanisms now allow a single physical dataset to be read as more than one format. Delta UniForm, described earlier, exposes Delta tables as Iceberg and supports the Iceberg REST catalog interface (Databricks, as of 2026). Hudi’s native Iceberg support lets a team manage a table with Hudi and serve it as Iceberg (Apache Hudi documentation, as of 2026). And Apache XTable — an incubating project backed by Microsoft, Google, and Onehouse — provides omni-directional metadata translation between all three formats: any format to any format, with no copying of the underlying data (Dremio; Onehouse, as of 2026). XTable works by generating the metadata that each target format expects, pointing it at the same Parquet files, so the cost of translation is metadata generation rather than data duplication.

The cumulative effect of these developments is that Apache Iceberg has become the de-facto common denominator. It is the format that the other two can be exposed as, the one with the broadest native reach, and the one adopted by every major cloud provider and query engine (RisingWave; Dremio; Onehouse, as of 2026). The reasons most often cited are the same throughout the literature: vendor-neutral governance, partition evolution, and the widest multi-engine support.

How to Choose in 2026

Given convergence, the practical decision reduces to selecting the format that best fits the write path and the operational center of gravity, while relying on interoperability to satisfy diverse readers. The decision tree below summarizes the guidance, and the discussion that follows expands on each branch.

Iceberg as the safe default

For a new deployment that aims to remain vendor-neutral and to be queried by several engines, Iceberg is the safe default. It carries the lowest risk of lock-in, offers partition evolution and hidden partitioning, and is natively supported across the widest range of engines and warehouses. Choosing Iceberg also aligns with the direction of the ecosystem, since the other formats can be exposed as Iceberg but the reverse arrangement is less central to current tooling. For teams running engines on Kubernetes, where portability is already a goal, the neutrality argument extends naturally to the storage layer; the operational considerations of running such engines are discussed in the guide to database connections from Kubernetes pods.

Delta Lake for Databricks-centric stacks

When the platform is built around Databricks and Spark, Delta Lake remains the natural choice. Its integration is the deepest available, its tooling and governance through Unity Catalog are mature, and UniForm now mitigates the historical drawback of limited external reach by exposing the same tables as Iceberg. A team already invested in Databricks gains little by writing Hudi or Iceberg directly and may give up integration depth by doing so.

Hudi for streaming upserts

For ingestion dominated by high-frequency upserts and change-data-capture streams, Hudi remains the strongest fit. Its record-level indexing, Merge-on-Read table type, and built-in compaction and cleaning were designed for exactly this pattern, and its Hudi Streamer utility provides a tested ingestion path. The native Iceberg output then allows the same data to be served to analytical consumers as Iceberg, combining Hudi’s write strengths with Iceberg’s read reach.

When to rely on interoperability instead

Some organizations do not need to pick a single winner. When one dataset must serve consumers that expect different formats — for example, a Spark-on-Databricks team reading Delta and a Trino team reading Iceberg — relying on XTable or UniForm to translate metadata is often preferable to maintaining duplicate copies of the data. The decision then shifts from “which format” to “which write path produces the data, and which translations are needed for the readers.” This framing is the clearest sign of how far the field has moved from the original format competition.

Frequently Asked Questions

What is a lakehouse table format, and how does it differ from a file format like Parquet?

A file format such as Parquet defines how the bytes of a single data file are laid out. A table format is a metadata layer above many such files that presents them as one logical table with ACID transactions, schema evolution, and time travel. Parquet stores the data; the table format records which files belong to which committed version of the table and how to read them consistently.

Is Apache Iceberg replacing Delta Lake and Hudi?

Not exactly. Iceberg has become the de-facto common denominator that the other formats translate toward, and it is the safe default for new neutral deployments. Delta Lake and Hudi retain distinct strengths — deepest Spark integration and the most mature streaming-upsert tooling, respectively — and both can now expose their tables as Iceberg, so they continue to be used as write paths even where Iceberg is the serving format.

What is the difference between Hudi Copy-on-Write and Merge-on-Read?

Copy-on-Write rewrites the base file whenever a record is updated, which makes reads fast and writes heavier; it suits read-heavy tables with moderate update rates. Merge-on-Read appends updates to delta log files and merges them with the base file at query time, which makes writes light and low-latency; it suits high-frequency upserts, with background compaction periodically consolidating the deltas.

Does choosing one format lock an organization into one vendor or engine?

Less than it once did. Delta UniForm exposes Delta tables as Iceberg through the Iceberg REST catalog, Hudi can output Iceberg-compatible tables, and Apache XTable translates metadata among all three with no data copying. Iceberg in particular is vendor-neutral by design. Lock-in is now mainly a function of operational tooling and the write path rather than the storage format itself.

What did the Databricks acquisition of Tabular mean for the format landscape?

Databricks, the company behind Delta Lake, acquired Tabular — the startup founded by Iceberg’s original creators — for more than one billion dollars, with the deal revealed on June 4, 2024. Bringing Iceberg’s architects inside the Delta vendor signaled support for both formats and is widely read as the point at which the industry pivoted from a format competition toward interoperability.

References

• Apache Iceberg — official documentation
• Delta Lake — official site and documentation
• Apache Hudi — official documentation
• Apache XTable (incubating) — omni-directional table-format interoperability
• Databricks — acquisition of Tabular (announced June 4, 2024)

Comparative characterizations of engine reach and format maturity reflect 2026 guidance from RisingWave, Dremio, Onehouse, and lakeFS; figures and dated claims are attributed inline.

Conclusion

The lakehouse table format is the layer that turns object storage into a transactional database surface, and the three principal implementations express three design priorities: Iceberg’s engine-neutral interoperability, Delta Lake’s depth within the Spark and Databricks ecosystem, and Hudi’s strength in high-frequency streaming upserts. Their core guarantees converge, but their metadata models and operational characters remain distinct enough to matter on the write path.

The decisive shift of the past two years is that selecting a format no longer means accepting permanent lock-in. The Databricks–Tabular acquisition, Delta UniForm, Hudi’s Iceberg output, and Apache XTable have made it possible for one physical dataset to be presented as more than one format, with Iceberg emerging as the de-facto common denominator. The practical recommendation that follows is straightforward: default to Iceberg for new, multi-engine, vendor-neutral deployments; choose Delta Lake within Databricks-centric stacks; choose Hudi for upsert-heavy streaming ingestion; and lean on interoperability layers when one dataset must serve consumers expecting different formats. The right choice is the one that fits the write path, because interoperability can serve the readers afterward.

This article examines the architecture of Kubernetes pods and explains why directly connecting an external client to a database running inside a pod is more involved than the equivalent task with a standalone Docker container. The discussion is grounded in the networking model that Kubernetes uses and in the Service abstraction that the model requires. A common experience for engineers new to Kubernetes is that kubectl exec into a Postgres pod followed by psql -h localhost works as expected, while a parallel attempt from a developer laptop, using the pod IP reported by kubectl get pods -o wide, times out with no error. The credentials, the database, and the apparent network are the same, yet the second connection never completes. This outcome is not the result of a defect; it is a direct consequence of how the platform is designed, and understanding the design is the first step toward connecting to in-cluster databases in a reliable manner.

Why Kubernetes Exists in the First Place

Docker addressed a genuine packaging problem by allowing an application and its dependencies to be shipped as a single reproducible image that behaves consistently on a developer laptop, a continuous integration runner, and a production virtual machine. Readers who have not yet worked through the container model will find the Docker containers, from dev to production guide a useful prerequisite for the material that follows. Once an organization operates several dozen containers distributed across several dozen servers, however, Docker alone becomes insufficient. Several questions arise that the single-host model does not answer: which host should run a given container, what should happen if that host fails outside business hours, how can a new version be rolled out without dropping in-flight requests, how do containers on one host discover containers on another, and how should compute and memory budgets be enforced.

Kubernetes is the answer that became the industry consensus. It originated inside Google as a re-implementation of the Borg system and was open-sourced in 2014. Kubernetes is best understood as a cluster operating system: the operator declares the desired state of the workload, and a chain of controllers continuously reconciles the actual state of the cluster with that declaration. The unit that Kubernetes manages is not a container directly but a pod, a wrapper around one or more tightly coupled containers that share a network identity and storage. All higher-level objects, including Deployments, Services, StatefulSets, Jobs, and CronJobs, are abstractions that ultimately specify which pods should exist, where they should run, and how they should be exposed.

The control plane performs the coordination function, while worker nodes execute the workload. Each worker runs three components in its base layer. The kubelet is the agent that receives instructions from the API server and applies them to the local node. The container runtime, which is now almost always containerd or CRI-O, executes the containers themselves; Docker as a runtime was deprecated in version 1.20 and removed in version 1.24. The kube-proxy process programs the kernel iptables or IPVS rules that route Service IPs to actual pod endpoints. Pods are scheduled on top of these three components.

The Pod: Smaller Than a VM, Bigger Than a Container

A pod is the smallest deployable unit in Kubernetes. The simplest pod runs a single container, but the abstraction exists precisely because the smallest unit that an engineer sometimes needs to ship is more than one container. A common configuration places a primary application container alongside a sidecar container that handles logging, TLS termination, metric scraping, or database proxying. All containers in the same pod share a single network namespace, which means they can communicate over localhost, and they can share filesystem volumes. They are always scheduled together onto the same node, and their lifecycles are linked: they are created together, restarted together, and terminated together.

The manifest below shows the minimum specification for a pod. Three details are worth noting. The apiVersion: v1 field indicates that pods are part of the core Kubernetes API. The specification contains a single container running an Nginx image. There is no top-level restart policy, which reflects the fact that bare pods are not self-healing: when a node fails, the pod fails with it. Bare pods are therefore rarely used in production. They are useful primarily for one-off tests and as a teaching device.

# pod.yaml — the simplest possible pod
apiVersion: v1
kind: Pod
metadata:
  name: hello-pod
  labels:
    app: hello
spec:
  containers:
  - name: web
    image: nginx:1.27
    ports:
    - containerPort: 80
    resources:
      requests:
        cpu: "50m"
        memory: "64Mi"
      limits:
        cpu: "200m"
        memory: "128Mi"

The object that an operator actually deploys is a Deployment, a controller that maintains a desired number of identical pods, handles rolling updates, and recreates pods when nodes fail. The Deployment owns a ReplicaSet, which in turn owns the pods. Operators rarely reference pods directly. Instead, they reference the Deployment, and Kubernetes manages the underlying pods.

# deployment.yaml — a real workload
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  labels:
    app: api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
      - name: api
        image: ghcr.io/acme/api:1.4.2
        ports:
        - containerPort: 8000
        readinessProbe:
          httpGet: { path: /healthz, port: 8000 }
          initialDelaySeconds: 5
        livenessProbe:
          httpGet: { path: /livez, port: 8000 }
          initialDelaySeconds: 15
        env:
        - name: DB_HOST
          value: "postgres.db.svc.cluster.local"
        - name: DB_PORT
          value: "5432"

The Flat Networking Model and Its Implications

The Kubernetes networking model rests on four rules that appear simple on the surface but carry substantial implications for how traffic flows inside a cluster:

1. Every pod gets its own IP address, drawn from a cluster-wide CIDR range that does not overlap with the node IPs.
2. Pods on the same node can communicate without NAT.
3. Pods on different nodes can communicate without NAT.
4. The IP a pod sees as its own is the same IP that other pods see when they talk to it.

The fourth point carries the most weight. In a typical single-host Docker configuration, a container holds a private IP on a bridge network, and outbound traffic is translated through the host using NAT. Kubernetes deliberately avoids this arrangement. Every pod is a first-class participant in a single flat network, regardless of the physical machine that hosts it. The component that implements this property is a CNI plugin (Container Network Interface), of which several mature implementations exist, including Calico, Cilium, Flannel, Weave, the AWS VPC CNI, and the native plugin used by GKE. These plugins implement the same contract but differ in their mechanisms, which range from overlays based on VXLAN tunnels, to BGP route advertisement, to native cloud routing, to eBPF-based data planes.

This flat addressing scheme is convenient for application code, which simply connects to an IP address and proceeds, but it is demanding for operators who must reason about traffic flows. Every pod is mutually addressable inside the cluster, which means that in the absence of explicit policies, every pod is able to reach every database, every cache, and every internal API. This property becomes important in a later section, which explains why directly addressing a database pod is more fragile than it appears.

Services: How Pods Actually Find Each Other

Because pod IPs change on every restart, they cannot appear in a connection string. The Kubernetes solution to this problem is a Service, a stable virtual IP and DNS name that load-balances traffic to a set of pods identified by labels. Individual pods come and go, but the Service remains. Inside the cluster, every Service automatically receives a DNS name of the form service-name.namespace.svc.cluster.local, which is resolved by CoreDNS, the built-in DNS resolver of the cluster.

The manifest below defines a ClusterIP Service for a Postgres pod. The selector field is worth attention, because the Service matches by labels rather than by name. Any pod that carries the label app: postgres in the same namespace automatically becomes a backend.

# service.yaml — ClusterIP for Postgres
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: db
spec:
  type: ClusterIP            # default, can omit
  selector:
    app: postgres
  ports:
  - name: pg
    port: 5432              # the Service port
    targetPort: 5432        # the container port
    protocol: TCP

From any other pod inside the cluster, the command psql -h postgres.db.svc.cluster.local -p 5432 will succeed. The same command issued from a developer laptop will hang indefinitely. The next section examines the reasons for this gap in detail.

Why Direct Connections to a Database Pod Fail

The assumption that breaks down for engineers coming from a plain Docker workflow is the idea that a container can be reached as long as one knows its IP address and port. In Kubernetes, almost every element of that assumption is incorrect. The pod IP is real but private, ephemeral, and exists only on a network shared between the nodes of a single cluster. The container port is open inside the container but is not automatically exposed at any higher layer. There is no host-level port-publishing equivalent to docker run -p 5432:5432; the field hostPort exists but is discouraged for production use. The following paragraphs examine each failure mode in turn.

Pod IPs are ephemeral. The moment a pod restarts, for any reason ranging from a node reboot, to a failed liveness probe, to a manual kubectl rollout restart, to an eviction by the scheduler, the new pod receives a new IP address from the address pool that the CNI manages. Any client that retains a reference to the previous IP is now communicating with nothing, or in the worst case, with whatever pod has been allocated the recycled address. This is the reason pod IPs should never be written into a configuration file. The correct address to record is a Service DNS name, which CoreDNS resolves at lookup time.

The ClusterIP is not visible from outside the cluster. The Service IP that kubectl get svc reports, such as 10.96.42.7 in the earlier example, is a virtual IP. It does not belong to any physical or virtual network interface and exists only as an entry in the iptables tables that kube-proxy maintains on each node. A laptop outside the cluster has no route to the Service CIDR 10.96.0.0/12, and even a statically added route would not help, because no kernel outside the cluster contains the rules required to translate that virtual address.

Pods do not use the host network by default. Setting hostNetwork: true on a pod causes the container to share the network namespace of the node, with the consequence that the container port maps directly to a port on the node. This configuration is used by CNI agents, node-exporter, and similar infrastructure components. Applying it to a database, however, is poor practice: IP isolation is lost, port collisions become possible, and any node failure takes the database with it, since the address is tied to a specific host and cannot be moved.

NetworkPolicies can explicitly deny traffic. When the cluster runs a CNI that supports NetworkPolicy, which most modern plugins do, operators can write rules such as “only pods labeled role: api in the app namespace may connect to pods labeled app: postgres in the db namespace on port 5432.” When a default-deny baseline is in place and no allow rule has been written, all traffic is dropped. When no policies are present at all, all traffic is permitted, which presents its own security concerns.

# networkpolicy.yaml — only the api can talk to postgres
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: postgres-allow-api
  namespace: db
spec:
  podSelector:
    matchLabels:
      app: postgres
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: app
      podSelector:
        matchLabels:
          role: api
    ports:
    - protocol: TCP
      port: 5432

The container port is not automatically exposed at the node level. Docker users are accustomed to -p 5432:5432, which binds a host port to a container port. Kubernetes provides no equivalent automatic mapping. The containerPort field in a pod specification is documentation: it informs operators and tooling that the container intends to listen on the indicated port, but it does not open a path through any higher layer. External reachability requires a Service of the appropriate type and, in cloud environments, a security group rule that permits traffic to whichever node port the cloud load balancer or NodePort uses.

Databases are stateful, and stateful pods require stateful controllers. A plain Deployment treats its pods as interchangeable replicas. The Deployment will reschedule postgres-0 from node 2 to node 5 when node 2 becomes unhealthy, mounting whichever PersistentVolume is available, or no volume at all if the PersistentVolumeClaim has been deleted. A database instead requires a StatefulSet, which assigns each pod a stable identity such as postgres-0 or postgres-1, a stable per-pod DNS name served by a headless Service, and a stable PersistentVolumeClaim that remains attached to the same ordinal across reschedules. A misconfiguration in this area is a common cause of data loss for teams new to running databases on Kubernetes.

The request path is long, and any single weak link breaks it. When an external client reaches a pod, the request typically traverses the following sequence: client, public DNS, cloud load balancer, node IP, iptables DNAT, pod IP, container port, Postgres listener, pg_hba.conf check, and finally authentication. A misconfiguration at any stage, such as an incorrect TLS certificate, a security group blocking the load balancer health check, a pg_hba.conf rule that denies the source CIDR, or a Postgres listener bound to 127.0.0.1 inside the container rather than 0.0.0.0, produces a connection failure that appears identical to a network problem from the perspective of the client.

Three Reliable Connection Patterns

Three legitimate patterns exist for connecting a client to a database that runs in a pod, and the appropriate choice depends primarily on the location of the client. Selecting among them is largely a question of which client requires the connection and for how long.

Pattern A: In-cluster application to in-cluster database

This pattern is the default and the most reliable choice. The application pod sets DB_HOST=postgres.db.svc.cluster.local as an environment variable and opens a connection. CoreDNS resolves the name, kube-proxy translates the virtual IP into the address of a real pod through DNAT, and the connection succeeds. Pod restarts on either side remain transparent because every endpoint is named rather than pinned to a specific IP. This is also the pattern that Airflow workloads adopt when they run with the KubernetesExecutor described in the Apache Airflow data pipeline orchestration guide, in which each task is launched as a pod that reaches the database through a Service. The same pattern applies to dbt jobs running on Kubernetes and to Kafka consumer workloads running in pods.

Pattern B: Local developer to in-cluster database

The command kubectl port-forward opens a tunnel from a local port on a developer machine, through the Kubernetes API server, to a port on a pod. It is intended for development and one-off administrative tasks. The example below uses it against the headless Service that the next subsection defines:

# forward localhost:5432 to the postgres-0 pod's port 5432
kubectl port-forward -n db pod/postgres-0 5432:5432

# Or forward through the headless Service to whichever endpoint is selected
kubectl port-forward -n db svc/postgres 5432:5432

# Now from another terminal, on your laptop:
psql -h localhost -p 5432 -U app -d production

The Python client below connects through the forwarded port. The connection string specifies localhost, which is correct on the developer laptop. Inside the cluster, the same code would instead specify postgres.db.svc.cluster.local.

# dev_query.py — assumes "kubectl port-forward" is running
import os
import psycopg2
from psycopg2.extras import RealDictCursor

# Local dev: connect through kubectl port-forward
# In production (in-cluster), DB_HOST would be postgres.db.svc.cluster.local
DB_HOST = os.environ.get("DB_HOST", "localhost")
DB_PORT = int(os.environ.get("DB_PORT", "5432"))
DB_NAME = os.environ.get("DB_NAME", "production")
DB_USER = os.environ.get("DB_USER", "app")
DB_PASS = os.environ["DB_PASS"]  # required, no default

def fetch_recent_orders(limit: int = 50):
    """Read the most recent orders — example dev-time query."""
    with psycopg2.connect(
        host=DB_HOST,
        port=DB_PORT,
        dbname=DB_NAME,
        user=DB_USER,
        password=DB_PASS,
        connect_timeout=5,
        sslmode="require",   # still enforce TLS even on port-forward
    ) as conn:
        with conn.cursor(cursor_factory=RealDictCursor) as cur:
            cur.execute(
                "SELECT id, customer_id, total_cents, created_at "
                "FROM orders ORDER BY created_at DESC LIMIT %s",
                (limit,),
            )
            return cur.fetchall()

if __name__ == "__main__":
    rows = fetch_recent_orders()
    for row in rows:
        print(row)

Pattern C: External application to in-cluster database

This is the pattern about which most teams should hesitate. When an application outside the cluster needs to read from or write to the database, the preferred architecture is almost always to expose an API over HTTP or gRPC through an Ingress with TLS and authentication, and to let the API mediate access to the database. Legitimate cases for direct external access nevertheless exist, including analytics tools, business intelligence dashboards, and replication to external systems. In those cases the pattern takes the following shape: a Service of type LoadBalancer backed by the database pods, fronted by a cloud network load balancer, with the security group restricted to specific client CIDRs, mandatory TLS, and a credential rotation policy. When a managed database such as Amazon RDS, Google Cloud SQL, or Aurora can be substituted, that option is usually preferable. Operating Postgres inside Kubernetes is technically feasible, but it represents a significant operational commitment.

The StatefulSet plus headless Service pattern

A headless Service is the object produced when clusterIP: None is set in the specification. Rather than allocating a virtual IP, this configuration produces DNS A records, with one record per pod backend. When combined with a StatefulSet, the result is a set of stable per-pod hostnames, such as postgres-0.postgres.db.svc.cluster.local and postgres-1.postgres.db.svc.cluster.local. This naming arrangement is precisely what a primary-replica database deployment requires. The application directs writes to the hostname of the primary and reads to the hostname of any replica.

# headless service + statefulset for postgres
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: db
  labels:
    app: postgres
spec:
  clusterIP: None          # headless — no virtual IP
  selector:
    app: postgres
  ports:
  - name: pg
    port: 5432
    targetPort: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: db
spec:
  serviceName: postgres    # MUST match the headless Service name
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      terminationGracePeriodSeconds: 30
      containers:
      - name: postgres
        image: postgres:16.3
        ports:
        - containerPort: 5432
          name: pg
        env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: password
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
        readinessProbe:
          exec:
            command: ["pg_isready", "-U", "postgres"]
          initialDelaySeconds: 10
          periodSeconds: 5
        resources:
          requests:
            cpu: "500m"
            memory: "1Gi"
          limits:
            cpu: "2"
            memory: "4Gi"
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: gp3-ssd
      resources:
        requests:
          storage: 200Gi

Production databases almost always benefit from a purpose-built operator layered on top of this scaffolding, such as CloudNativePG, the postgres-operator developed by Zalando, or Crunchy PGO. These operators handle primary election, streaming replication, backups, point-in-time recovery, and rolling minor-version upgrades. Selecting an appropriate database backend is a separate concern; the database comparison for preprocessed time-series data serves as a useful companion reference for that decision.

Kubernetes 1.36 and What Changed in 2026

Kubernetes 1.36 is the most recent minor release as of this writing in May 2026, and it continues the project’s emphasis on stronger security defaults and on first-class support for AI workloads. According to the official release page (Source: kubernetes.io/releases, as of 2026-05-20), the project actively maintains release branches for the three most recent minor versions, currently 1.34, 1.35, and 1.36. Version 1.33 entered maintenance on 2026-04-28 and reaches end of life on 2026-06-28. The release cadence is rapid enough that operators running anything older than 1.33 are already outside the supported window.

The promotion of User Namespaces to general availability is the most prominent security change in 1.36. When user namespaces are enabled, the root user inside a container is mapped to an unprivileged user on the host. This arrangement substantially reduces the impact of a container escape: even when an attacker compromises a container running as UID 0, they emerge on the host as a high-numbered unprivileged user, such as UID 100000, with no special privileges. For database pods specifically, a compromised Postgres container no longer translates directly into root access on the node. In combination with seccomp and AppArmor profiles, this change closes one of the long-standing gaps between Kubernetes security and traditional virtual machine isolation.

Mutating Admission Policies, also promoted to general availability, bring declarative mutations expressed in the Common Expression Language (CEL) to the admission chain, replacing many uses of webhook-based mutating admission controllers. Operators can now write policies that, for example, automatically inject sidecar containers, attach labels, set default resource requests, or enforce image-registry rules, without operating a separate webhook server. The result is less infrastructure to maintain and fewer failure modes when a webhook becomes unavailable.

Fine-Grained Kubelet API Authorization, now generally available, allows the kubelet to enforce per-verb RBAC on its own API rather than treating all operations uniformly. This change matters for hardening: tools that require nodes/proxy can be restricted to read-only operations, and the kubelet can refuse risky combinations that previously required cluster-admin privileges in order to be fully restricted.

Beyond security, version 1.36 continues to invest in AI workload support. It introduces refinements to Dynamic Resource Allocation (DRA) for GPU scheduling, adds support for accelerator partitioning, and improves the ability of the scheduler to handle long-running training jobs alongside short-lived inference pods. The trajectory is clear: the pattern of Kubernetes as an AI platform, which grew rapidly in 2024 and 2025 as model-serving workloads migrated off bespoke infrastructure, has been a first-class concern for two consecutive release cycles. For language and runtime choices when developing operators or controllers around these new APIs, the Python and Rust comparison provides a useful framing. The controller-runtime ecosystem in Go remains dominant, but Rust-based operators are gaining ground for performance-sensitive components.

Frequently Asked Questions

Can a pod have more than one container?

Yes, and it is a common design pattern. The most frequent reason is the sidecar — a helper container that does logging, TLS termination, service-mesh proxying (Envoy in Istio or Linkerd), or connection pooling. All containers in a pod share a network namespace and can share volumes, but they remain separate processes with separate filesystems. Use multiple containers when their lifecycles are genuinely coupled. If the answer to “can these scale independently?” is yes, they belong in separate pods.

Why not just expose every database pod with a NodePort and connect directly?

NodePort opens the same port on every node in the cluster, in the 30000–32767 range, and routes it to whichever pod backs the Service. Three problems: the port numbers are non-standard so client tooling fights you, every node becomes an attack surface for the database, and you still need a cloud security group or firewall rule to control who can hit those ports. NodePort is fine for on-prem clusters without a cloud LB or for very specific debug scenarios. It is not a substitute for proper Service architecture.

Is kubectl port-forward safe to use in production?

It is safe to use, but it should not be how production traffic flows. The tunnel runs through the API server and consumes API-server resources. It bypasses NetworkPolicy — if you can port-forward, you can connect, regardless of how strict your in-cluster policies are. RBAC controls who can use it, and you should treat pods/portforward on a database namespace as a sensitive verb subject to audit. For production traffic, use a real Service.

What is the difference between a StatefulSet and a Deployment?

A Deployment treats pods as interchangeable. It will scale up by spinning up new pods with random suffix names, scale down by killing any of them, and roll updates in parallel. A StatefulSet maintains ordered, named pods (name-0, name-1, name-2) that always come up in order, always shut down in reverse order, and each get their own stable PersistentVolumeClaim. Use Deployment for stateless apps. Use StatefulSet for anything that has identity — databases, message brokers, ZooKeeper, distributed coordination services. Kafka brokers running in Kubernetes are a textbook StatefulSet workload.

Should I actually run my database in Kubernetes, or use a managed service?

For most teams below the scale of needing a database engineer on the org chart, managed (RDS, Cloud SQL, Aurora, AlloyDB, Spanner) is the right answer. Operating a stateful workload well — backups, point-in-time recovery, minor-version upgrades, failover, performance tuning, observability — is a continuous engineering investment that managed services amortize across thousands of customers. Run databases in your cluster when you have a real reason: cost at scale, regulatory data residency, latency requirements that make a separate database tier unworkable, or a database that managed offerings do not provide. The operator ecosystem (CloudNativePG and friends) makes this much more tractable than it was five years ago, but it is still real work.

Related Reading

References

• Kubernetes releases — official release and support timeline
• Kubernetes cluster networking concepts
• Kubernetes 1.36 sneak peek (official blog, March 2026)
• Connecting applications with Services (official tutorial)
• InfoQ: Kubernetes 1.36 released (May 2026)

Conclusion

Connecting to a database that runs in a Kubernetes pod feels harder than it should because Kubernetes is solving a different problem than many engineers initially assume. It is not an elaborate replacement for docker run. It is a cluster operating system whose entire networking model is designed around the principle that pods communicate with other pods through stable abstractions, and external clients reach applications through carefully chosen entry points. The pod IP revealed by kubectl get pods -o wide is a debugging convenience rather than an address suitable for client traffic. The ClusterIP shown by kubectl get svc is a virtual construct held together by iptables rules. The correct address for production traffic originating inside the cluster is a DNS name served by CoreDNS and backed by a Service whose membership the controllers maintain. The correct address from outside the cluster is whatever the LoadBalancer, Ingress, or bastion-host configuration specifies, and it is never a pod IP.

Three points are worth retaining from this discussion. First, kubectl port-forward is well suited to development workflows and unsuited to production traffic. Second, stateful workloads require a StatefulSet, a PersistentVolumeClaim, and a headless Service in combination, or data loss is likely. Third, in Kubernetes 1.36 and beyond, security defaults are tightening, with User Namespaces reaching general availability as the most consequential change, which benefits anyone running databases in pods. Even with these improvements, however, the number of ways in which a connection between an external client and an in-cluster database can fail remains large enough that exposing Postgres directly to the public internet is almost always inferior to placing an API tier in front of the database. The recommended approach is to build the conservative, layered version first, and to reserve more aggressive shortcuts for cases that genuinely warrant them.