Apache Parquet and Apache Arrow Internals: Columnar Storage, Encoding, and Predicate Pushdown

Apache Parquet and Apache Arrow are the two most widely deployed columnar data formats in modern analytics, yet most engineers who depend on them daily treat them as opaque. A query runs against a Parquet file through Spark, DuckDB, Polars, or pandas, the result returns quickly, and the reasons remain hidden. This post opens the black box. It describes what the bytes of a Parquet file actually look like on disk, how encoding and compression stack into a layered pipeline, how metadata and statistics let a reader skip most of the data it never needs, and how Apache Arrow provides the complementary in-memory format that lets independent engines exchange columns without paying a serialization cost. The goal is a mechanistic understanding that translates directly into tuning decisions: row-group sizing, encoding choice, compression codec, page index, and Bloom filters.

The two projects are designed to work together. Parquet is a space-efficient format optimized for durable storage on disk or in object storage. Arrow is a compute-efficient format optimized for processing in memory. A reader decodes Parquet pages into Arrow arrays, operates on them in a CPU-friendly layout, and hands them to another library through raw memory pointers. Understanding both halves, and the bridge between them, explains why a column-oriented analytical stack behaves the way it does.

Why columnar storage exists

A data file must choose an order in which to write the values of a table. A row-oriented format stores all fields of the first record together, then all fields of the second record, and so on. This layout suits transactional workloads that read or write entire records one at a time. An analytical workload behaves differently. It scans a small number of columns across a very large number of rows, computing aggregates, filters, and joins. For such a query, a row-oriented file forces the reader to touch every field of every record even when only two of forty columns are relevant.

A column-oriented, or columnar, format inverts the layout. It stores all values of the first column together, then all values of the second column, and so on. A query that reads two columns reads only those two contiguous regions and ignores the rest of the file. This property, reading only the requested columns, is called projection pushdown, and it is essentially free in a columnar format because the requested data is already grouped.

Columnar layout produces a second benefit that compounds the first. Values within a single column share a data type and often share statistical structure — a timestamp column tends to increase monotonically, a category column repeats a small set of strings, a sensor column clusters around a mean. Grouping like values together lets the format apply type-aware encodings and lets general-purpose compressors find far more redundancy than they could in an interleaved row layout. The figure below contrasts the two layouts for a small table.

This division of labor is deliberate. A durable storage format optimizes for size, because bytes on disk or in object storage cost money and network transfer time. An in-memory format optimizes for processing speed, because a CPU or GPU benefits from predictable, contiguous, aligned buffers. Apache Parquet fills the first role and Apache Arrow fills the second. The remainder of this post examines each in turn, then the bridge that connects them.

Anatomy of a Parquet file on disk

A Parquet file is a self-describing container built from a strict hierarchy. Understanding that hierarchy is the foundation for every tuning decision that follows, because each level maps to a unit of I/O or a unit of metadata.

The file, row group, column chunk, and page

At the top is the file. A file is divided horizontally into row groups. A row group is a partition of the rows — a contiguous band of records — and it is the largest unit that can be processed independently. Within a row group, each column is stored as a column chunk, which holds every value of that one column for that band of rows. Each column chunk is in turn divided into pages. A page is the smallest unit that must be read and decompressed in full to access any record inside it, and each page is encoded and compressed independently. The default row group size is 128 MB (the parquet.block.size configuration is 134217728 bytes) and the default page size is 1 MB (the parquet.page.size configuration is 1048576 bytes), according to the Parquet configuration documentation (parquet.apache.org, as of 2026-07-02).

The file is framed by a four-byte magic value, PAR1, at both the beginning and the end. The trailing region before the closing magic is the footer, which holds the FileMetaData: the schema, the list of row groups, the metadata and statistics for every column chunk, and the byte offsets that locate optional structures such as the Page Index and Bloom filters. A reader opens a Parquet file by seeking to the end, reading the footer, and using it as a map to the rest of the file. The figure below shows this layout.

The Parquet specification defines this structure in two parts that are read together: a prose format document and a Thrift definition file, parquet.thrift, that specifies the exact metadata structures (apache/parquet-format README, as of 2026-07-02). The latest format specification version is 2.12.0 (apache/parquet-format CHANGES.md, as of 2026-07-02). The format continues to evolve — a Variant type for semi-structured data was announced in February 2026 (parquet.apache.org blog, as of 2026-07-02).

Data page versions

Pages come in several types. A DictionaryPage stores the distinct values of a dictionary-encoded column. A DataPage carries the actual values. There are two versions of the data page. The original DataPage (V1) stores repetition levels, definition levels, and encoded values together in one compressed block. DataPageV2 separates the repetition and definition levels from the values, allows those levels to remain uncompressed while the values are compressed, and gives per-page control over whether compression is applied. This separation is what unlocks the delta encoding family described in the next section. An IndexPage type is reserved in the specification. The figure below contrasts the two data page versions.

Repetition and definition levels are the mechanism by which Parquet represents nested and repeated fields in a flat columnar store. This technique originates in the Dremel system described by Melnik and colleagues (VLDB 2010), the foundational paper for Parquet’s nested encoding. A definition level records how many of a column’s optional or nested ancestors are actually present for a given value, which is how nulls at any nesting depth are represented without storing placeholder values. A repetition level records at which nesting level a repeated field begins a new list. Together the two levels let a reader reconstruct arbitrarily nested records from flat columns.

Encoding and compression

Two distinct byte-reduction layers operate inside a Parquet column chunk, and conflating them is a common source of confusion. Encoding is a structural, format-aware transformation applied to the values of one column. Compression is a byte-level, general-purpose transformation applied to the already-encoded bytes of a page. The two stack: values are first encoded, then the encoded output is compressed. A reader reverses the order, decompressing a page and then decoding its values.

The encoding families

PLAIN encoding is the baseline. It writes values in their natural fixed-width or length-prefixed form with no attempt at reduction, and it exists as a fallback when no structural pattern can be exploited.

Dictionary encoding is the workhorse for columns with a limited number of distinct values, such as country codes, status strings, or category labels. The writer builds a dictionary of distinct values, stored in a DictionaryPage, and replaces each value in the data pages with a small integer index into that dictionary. Those indices are then written with the RLE_DICTIONARY encoding, which has enum value 8 in the specification, is valid for all physical types, and is the default for string columns (apache/parquet-format Encodings.md, as of 2026-07-02). If the dictionary grows beyond the configured dictionary page size — the default dictionary page size is 1 MB (parquet.apache.org configurations, as of 2026-07-02) — the writer abandons the dictionary for that column chunk and falls back to PLAIN. One dictionary page is written per column per row group.

The RLE / bit-packing hybrid encoding underlies dictionary indices and also carries the repetition and definition levels. It combines run-length encoding, which represents a repeated value as a value-plus-count pair, with bit-packing, which stores small integers using only as many bits as their range requires rather than a full 32 or 64 bits. The encoder switches between the two modes according to which is smaller for each run.

The delta encoding family targets numeric and byte-array columns and is a DataPageV2 feature, not used for V1 pages (parquet.apache.org encodings, as of 2026-07-02). DELTA_BINARY_PACKED, enum value 5, applies to INT32 and INT64 columns and stores the differences between consecutive values rather than the values themselves (apache/parquet-format Encodings.md, as of 2026-07-02). For a sorted or monotonic column such as a timestamp or an auto-incrementing identifier, those differences are small and highly repetitive, so the encoded output is far smaller than the raw integers. DELTA_BYTE_ARRAY, enum value 7, applies to BYTE_ARRAY and FIXED_LEN_BYTE_ARRAY columns; it encodes the lengths of the byte arrays using DELTA_BINARY_PACKED and then concatenates the bytes, and it is preferred over PLAIN for byte arrays (parquet.apache.org encodings, as of 2026-07-02). A related variant, DELTA_LENGTH_BYTE_ARRAY, separates the length stream from the data stream for strings without shared prefixes.

BYTE_STREAM_SPLIT, enum value 9, applies to INT32, INT64, FLOAT, DOUBLE, and FIXED_LEN_BYTE_ARRAY columns (apache/parquet-format Encodings.md, as of 2026-07-02). It does not reduce size by itself. Instead it scatters the K bytes of each value into K separate streams, placing all the first bytes together, then all the second bytes, and so on. Floating-point values that cluster around a common magnitude share high-order bytes, and grouping those bytes together gives the downstream compressor much more redundancy to exploit (parquet.apache.org encodings, as of 2026-07-02). This is a clear illustration of the encoding-then-compression stack: the encoding rearranges bytes so that the compression layer performs better.

The compression codecs

After encoding, each page is compressed with a block-level, general-purpose codec. The supported codecs are UNCOMPRESSED, SNAPPY, GZIP, LZ4 and LZ4_RAW, ZSTD, and BROTLI. Because each page is compressed independently, a reader can decompress exactly the pages a query touches and no others. The choice among codecs is a trade-off between compression ratio and processing speed, and the appropriate choice is workload-dependent rather than fixed.

Metadata, statistics, and pushdown

The performance of an analytical scan depends less on how fast the reader processes data than on how much data it manages to avoid reading at all. Parquet supports this avoidance through metadata stored in the footer and in optional index structures. Two mechanisms operate together: projection pushdown selects which columns to read, and predicate pushdown selects which rows — more precisely, which row groups and pages — to read.

Projection pushdown was described earlier. Because each column is stored as an independent column chunk, a reader consults the footer to find the byte ranges of the requested columns and reads only those ranges. No special index is required; the columnar layout makes it free.

Predicate pushdown is more involved and operates in three pruning tiers, each addressing a case the previous tier cannot handle. The figure below shows the tiers as a funnel that progressively narrows the data a query must read.

The three pruning tiers

The first tier is row-group statistics. The footer stores, for each column chunk, the minimum value, the maximum value, and the null count. When a query filters on a column, the reader compares the predicate against these statistics. If a row group’s minimum and maximum for the filtered column cannot possibly satisfy the predicate, the reader skips the entire row group — up to 128 MB of data — without reading any of its pages. This tier is coarse but very cheap, since the statistics are already in the footer the reader parsed on open.

The second tier is the Page Index, which brings the same idea down to page granularity. The Page Index was introduced in Parquet 1.11 and is used by Spark 3.2.0 and later (CERN Databases blog and parquet-format PageIndex.md, as of 2026-07-02). It consists of two structures. The ColumnIndex stores the minimum and maximum values for each page in a column chunk, letting the reader identify exactly which pages could contain matching rows. The OffsetIndex maps a row index to the byte offset of the page that contains it, so once the ColumnIndex has selected the relevant pages, the reader seeks directly to them. Together these let a reader skip individual pages within a column chunk that survived the row-group filter.

The third tier is the Bloom filter, which handles a case the range-based tiers cannot. Minimum and maximum statistics prune well when the filtered values fall outside a range, but they are useless for equality predicates on high-cardinality columns whose values are scattered across the whole domain. A filter such as user_id = 90422 will match the min/max range of almost every row group even though only a handful of rows qualify. A Bloom filter is a compact probabilistic structure that answers set membership. Parquet supports exactly one representation, the Split Block Bloom Filter (SBBF), whose block size is 256 bits, arranged as eight contiguous 32-bit words (apache/parquet-format BloomFilter.md, as of 2026-07-02). It answers a membership query with either “definitely not present” or “probably present” — it produces no false negatives, only occasional false positives (apache/parquet-format BloomFilter.md, as of 2026-07-02). When the answer is “definitely not present,” the reader skips the row group. Dictionary pages can serve a similar membership-pruning role, since a value absent from a column chunk’s dictionary cannot appear in that chunk.

Apache Arrow and the Parquet-to-Arrow bridge

Parquet describes data at rest. Once a reader decodes a Parquet page, the values must live somewhere in memory in a form suitable for computation. That form is Apache Arrow. Arrow defines a language-independent, in-memory columnar layout for both flat and nested data, designed for efficient analytics on CPUs and GPUs (arrow.apache.org Columnar.html, as of 2026-07-02). The latest Apache Arrow release is 24.0.0, published on 2026-04-21, comprising 259 resolved issues and 325 commits from 57 contributors over roughly three months (arrow.apache.org blog, as of 2026-07-02).

Where Parquet optimizes for size on disk, Arrow optimizes for access speed in memory. An Arrow array stores a column’s values in a contiguous buffer with a separate validity bitmap indicating which entries are null. This layout is predictable and aligned, so a processor can iterate over it sequentially and apply vectorized instructions that operate on many values per cycle. The Parquet-to-Arrow bridge decodes Parquet pages directly into Arrow arrays: a library such as pyarrow reads a Parquet file and produces Arrow Tables, translating the on-disk encodings into the in-memory layout in one step.

Zero-copy and the C Data Interface

The most consequential property of Arrow is that independent libraries can agree on the exact byte layout of a column and therefore share it without copying. When a query engine, a dataframe library, and a machine-learning framework all represent a column as the same Arrow buffer, one can hand the column to another by passing a pointer rather than serializing, transmitting, and deserializing the data. This is the mechanism the Arrow C Data Interface formalizes. It enables zero-copy sharing between runtimes within a single process through memory pointers, and its only dependency is a stable C application binary interface (ABI) — a fixed set of C structures that every participant understands (arrow.apache.org CDataInterface.html, as of 2026-07-02).

This capability is why an in-process analytical stack can combine tools that were developed independently without incurring a translation tax between them. A query engine can execute a filter, hand the resulting Arrow columns to a dataframe library for a transformation, and pass those to a machine-learning framework, all without any of the three re-serializing the data. The comparison of in-process engines that read Parquet and exchange Arrow buffers is examined in the companion post on DuckDB versus Polars for in-process analytics.

Three ways Arrow data moves

Arrow data is exchanged through several mechanisms, and distinguishing them prevents confusion. The C Data Interface moves data within a single process by passing pointers to buffers already in memory. Arrow IPC, also known as the Feather file format, serializes Arrow data into a byte stream for transmission across a network or for writing to a file — it is a wire and storage format for Arrow itself. Parquet, finally, is the durable at-rest format optimized for long-term storage rather than for immediate computation. Each serves a different boundary: within a process, across a process or network, and at rest. The figure below maps the three mechanisms to the boundaries they cross.

Table formats such as Apache Iceberg, Delta Lake, and Apache Hudi layer a metadata and transaction management system on top of Parquet data files rather than replacing them; the distinction between the file layer and the table layer is examined in the comparison of Iceberg, Delta Lake, and Hudi table formats. A transformation tool that materializes analytical models frequently writes those models as Parquet, a pattern covered in the guide to dbt transformation pipelines.

Practical guidance for tuning

The internals described above translate into a small set of tuning decisions. None of them has a single correct answer; each depends on the storage medium, the reader’s memory budget, and the query patterns.

Row-group sizing

The default row-group size is 128 MB (parquet.apache.org configurations, as of 2026-07-02). A larger row group amortizes footer and metadata overhead across more rows and produces longer contiguous reads, which suits object storage where each request has latency. A smaller row group increases the granularity of row-group statistics pruning and reduces the memory a reader must hold to process one row group. When files are read from object storage, aligning the row-group size with the object store’s block or multipart size reduces the number of range requests. When a reader has a constrained memory budget, a smaller row group prevents it from having to buffer an entire 128 MB band per column.

Sorting to help encoding and pruning

Sorting the data by a frequently filtered column improves two mechanisms at once. First, a sorted numeric column has small, repetitive deltas, which makes DELTA_BINARY_PACKED encoding highly effective. Second, sorting tightens the minimum and maximum ranges within each row group and page, so that row-group statistics and the Page Index prune far more aggressively — an unsorted column tends to span the full value range in every row group, defeating range-based pruning. Sorting therefore benefits both file size and scan speed.

When to enable Bloom filters

A Bloom filter earns its space only on a high-cardinality column that is queried with equality predicates, such as a user identifier, a session token, or an email address. On such a column, minimum and maximum ranges overlap across every row group and cannot prune, so the Split Block Bloom Filter provides the only membership-based pruning available. On a low-cardinality column, a Bloom filter is wasteful, because dictionary encoding and range statistics already prune those columns effectively. Bloom filters should be enabled selectively, per column, based on query patterns.

Choosing a codec and page version

For hot data queried frequently, Snappy’s fast decompression favors query latency. For cold or archival data, Gzip’s higher ratio reduces storage cost at the expense of slower reads. Zstd offers a tunable middle ground and is increasingly chosen as a default because its ratio approaches Gzip’s at speeds closer to Snappy’s; the exact ratio remains workload-dependent and should be measured on the real dataset. DataPageV2 is required to use the delta encoding family, so writers that store sorted integer or timestamp columns benefit from enabling V2 pages. These file-level choices are typically set once in the writer configuration of the job that produces the data, whether that job runs under an orchestrator such as the one described in the guide to Apache Airflow pipeline orchestration or as part of an ingestion pipeline like the one in the walkthrough of an InfluxDB-to-Iceberg data pipeline.

Frequently Asked Questions

What is the difference between Apache Parquet and Apache Arrow?

Parquet is a format for storing columnar data on disk or in object storage, optimized for space efficiency through encoding and compression. Arrow is a format for holding columnar data in memory, optimized for fast processing on CPUs and GPUs. A reader decodes Parquet files into Arrow arrays to compute on them. The two are designed together: Parquet handles data at rest and Arrow handles data in memory.

What is the default row-group and page size in Parquet?

The default row-group size is 128 MB, set by the parquet.block.size configuration at 134217728 bytes, and the default page size is 1 MB, set by parquet.page.size at 1048576 bytes (parquet.apache.org configurations, as of 2026-07-02). A row group is a horizontal band of rows processed independently, and a page is the smallest unit that must be read and decompressed in full to access any record inside it.

How does predicate pushdown skip data in a Parquet file?

Predicate pushdown prunes data in three tiers. Row-group statistics in the footer store per-column min, max, and null count, letting a reader skip an entire row group. The Page Index refines this to individual pages using per-page min and max values with a row-index-to-offset map. The Split Block Bloom Filter handles equality on high-cardinality columns where ranges cannot prune, answering “definitely not present” or “probably present” (apache/parquet-format BloomFilter.md, as of 2026-07-02).

Why do Parquet files need both encoding and compression?

Encoding and compression are two separate layers that stack. Encoding is a structural, format-aware transformation of one column’s values, such as dictionary encoding or delta encoding, that exploits the column’s type and distribution. Compression is a general-purpose byte-level codec, such as Snappy or Zstd, applied to the already-encoded page. Encoding often rearranges bytes so the compressor finds more redundancy; BYTE_STREAM_SPLIT is a clear example, adding no size reduction itself but improving downstream compression (parquet.apache.org encodings, as of 2026-07-02).

What does the Arrow C Data Interface do?

The C Data Interface lets separate libraries in the same process share Arrow column buffers without copying, by passing pointers through a small set of standard C structures. Its only requirement is a stable C application binary interface (arrow.apache.org CDataInterface.html, as of 2026-07-02). This lets a query engine, a dataframe library, and a machine-learning framework exchange columns without serializing and deserializing the data at each handoff.

References

• Apache Parquet format specification and parquet.thrift — apache/parquet-format, spec version 2.12.0 (as of 2026-07-02)
• Parquet encodings reference — parquet.apache.org documentation (as of 2026-07-02)
• Parquet Split Block Bloom Filter specification — apache/parquet-format BloomFilter.md (as of 2026-07-02)
• Apache Arrow columnar format specification — arrow.apache.org (v24.0.0, as of 2026-07-02)
• Apache Arrow C Data Interface — arrow.apache.org (v24.0.0, as of 2026-07-02)
• Dremel: Interactive Analysis of Web-Scale Datasets — Melnik et al., VLDB 2010, the foundational source for Parquet’s repetition/definition-level nested encoding

Conclusion

A Parquet file is not an opaque blob but a carefully layered structure whose every level serves a purpose. The file-to-row-group-to-column-chunk-to-page hierarchy makes projection pushdown free and gives predicate pushdown its units of pruning. The encoding layer exploits the type and distribution of each column, and the separate compression layer removes the byte-level redundancy that remains. The footer’s statistics, the Page Index, and the Split Block Bloom Filter combine into a three-tier funnel that lets a reader avoid the majority of the data a query does not need. Apache Arrow completes the picture by defining the in-memory layout into which those pages decode, and its C Data Interface lets independent libraries share columns without paying a serialization cost.

These internals are not academic. Row-group sizing, sorting by a filtered column, selective Bloom filters, DataPageV2 for delta encodings, and codec choice are all decisions that follow directly from understanding the mechanism. An engineer who knows what the bytes look like can reason about why a scan is slow and what to change, rather than treating the format as a black box. The formats continue to evolve — Parquet at specification version 2.12.0 with a new Variant type announced in February 2026, and Arrow at release 24.0.0 from April 2026 (as of 2026-07-02) — but the columnar principles described here remain the stable foundation beneath every analytical query that touches them.