Parquet Compression Codecs Benchmarked: zstd, snappy, gzip, and lz4 on 10 Million Rows

By kongastral

Published July 19, 2026 · 20 min read

Why codec choice is a size-versus-speed decision

Storing the same 10,000,000-row table as Apache Parquet produced files ranging from 125.06 MiB with no compression down to 68.52 MiB with Zstandard at level 9 — a difference of roughly 45 percent in on-disk footprint from a single writer setting. The write cost, however, ranged from 0.7139 seconds to more than 44 seconds depending on the codec chosen. Those two numbers, measured on one machine over one dataset, frame the practical question this benchmark addresses: which Parquet compression codec to select, and why the answer is rarely the codec that produces the smallest file.

Apache Parquet is a columnar file format, meaning values from the same column are stored together rather than interleaved row by row. Compression is applied per column chunk after encoding, so the codec operates on runs of similar values, which is why columnar layout compresses far better than row-oriented storage. The format specification defines a fixed set of codecs — among them SNAPPY, GZIP, ZSTD, and LZ4 — and each represents a different point on the curve that trades computation for space. The mechanics of columnar layout, page encoding, and predicate pushdown are covered in a companion article on Apache Parquet and Apache Arrow internals; the focus here is narrower and empirical, measuring what those codecs actually cost and save on one representative dataset.

The reason a benchmark is worth running rather than reasoning from first principles is that the tradeoffs interact. A codec that compresses tightly can be slow enough to write that it stalls an ingestion pipeline. A codec that decompresses quickly may barely shrink the file. Row-group size, which controls how finely a query engine can skip data, changes compression ratio at the same time. Dictionary encoding, applied before the codec runs, can matter more than the codec itself for certain columns. Measuring these effects together, on the same data with the same tooling, is the only way to see which choices dominate.

Test environment: hardware, software, and dataset

Compression benchmarks are only interpretable when the machine, the library versions, and the data are stated exactly, because decompression speed is CPU-bound and codec implementations differ between library releases. The complete environment capture is recorded in the benchmark repository; the essentials are reproduced below.

Hardware and software

All writing and reading was done through the PyArrow Parquet writer, so the results reflect the Arrow C++ implementations of each codec at version 25.0.0. A different language binding or a different Arrow release could shift the absolute times, which is precisely why the versions are pinned and stated. The PyArrow write_table documentation lists the codec names accepted by this writer and confirms that its default codec is snappy.

The dataset

The benchmark reused a single 10,000,000-row table shared with a sibling study on DuckDB and Polars in-process analytics, so that both studies run on identical data. The table has five columns of mixed type, an in-memory Arrow footprint of 312,390,663 bytes (297.9 MiB), and the following schema:

• ts — timestamp[us], event time spanning roughly 90 days (November 2023 to February 2024).
• user_id — int32, values in the range 1 to 499,997.
• category — string, low cardinality with 10 distinct values (books, automotive, electronics, garden, beauty, apparel, home, grocery, and others).
• amount — float64, values from 0.09 to 3,714.81.
• flag — bool.

Two preparation steps were applied once, before any timing. First, the category column was decoded from its stored dictionary form to plain UTF-8 text, so that the writer’s dictionary flag — and not a carried-over Arrow dictionary — is the only thing deciding whether dictionary encoding is applied at the file level. Second, the table was sorted by ts ascending. Time-series data is realistically stored in time order, and sorting makes each row group hold a tight, non-overlapping range of timestamps, which is a precondition for the predicate-pushdown measurements described later.

Methodology: how every number was produced

Each measurement was repeated three times, and the median of the three runs is quoted throughout; the recorded results file also stores the full list of run times for inspection. Timings use a wall-clock performance counter, and garbage collection is forced before each timed operation to reduce interference.

One property of the setup deserves explicit statement rather than silent assumption: the read-cache state is warm. Reads happen immediately after each file is written, so the file is present in the operating-system page cache. Read timings therefore measure the CPU cost of decoding and parsing the columnar data, not cold disk input. This is the standard basis for comparing codecs because it isolates decompression speed from storage latency, but it means the read numbers should be read as decode cost, not as the end-to-end latency of a query against a cold object store. File sizes, by contrast, are exact on-disk byte counts, and per-column compressed and uncompressed sizes are read directly from the Parquet footer metadata.

Three separate experiments were run. The codec matrix fixed the row-group size at 1,000,000 rows and dictionary encoding on, then wrote and read the table with each of six codecs. The row-group study fixed the codec at Zstandard level 3 and varied the row-group size. The dictionary study fixed the codec at Zstandard level 3 and wrote the table twice, once with dictionary encoding on and once off, comparing the string column. The raw console output of the codec matrix is reproduced verbatim below.

==============================================================================
CODEC MATRIX  (row_group_size=1000000, dictionary encoding ON)
==============================================================================
predicate: ts >= 2024-02-08 10:13:18.975015  (expected ~498,711 rows, 4.99%)

[none    ] size=  125.06 MiB  write_med=0.7139s  read_med=0.0739s  col_med=0.0083s  filt_push_med=0.0176s (rows=498,711)  filt_nopush_med=0.0163s
[snappy  ] size=  102.17 MiB  write_med=0.8760s  read_med=0.0890s  col_med=0.0092s  filt_push_med=0.0170s (rows=498,711)  filt_nopush_med=0.0303s
[zstd_l3 ] size=   68.61 MiB  write_med=0.9421s  read_med=0.0783s  col_med=0.0140s  filt_push_med=0.0173s (rows=498,711)  filt_nopush_med=0.0406s
[zstd_l9 ] size=   68.52 MiB  write_med=2.3837s  read_med=0.0755s  col_med=0.0137s  filt_push_med=0.0171s (rows=498,711)  filt_nopush_med=0.0407s
[gzip    ] size=   74.90 MiB  write_med=44.3629s  read_med=0.1048s  col_med=0.0352s  filt_push_med=0.0230s (rows=498,711)  filt_nopush_med=0.0900s
[lz4     ] size=  102.65 MiB  write_med=0.8580s  read_med=0.0858s  col_med=0.0084s  filt_push_med=0.0172s (rows=498,711)  filt_nopush_med=0.0216s

The codec matrix: file size, write speed, read speed

The full codec matrix is shown in the table below. Write and read medians are in seconds; file size is the exact on-disk footprint in MiB; the single-column read measures reading only the amount column.

Three findings stand out. First, the two Zstandard settings produced almost the same file — 68.61 MiB at level 3 versus 68.52 MiB at level 9 — a difference of under 0.10 MiB, yet level 9 took 2.3837 s to write against 0.9421 s for level 3, roughly two and a half times longer. On this data, the higher level buys essentially nothing for a substantial write cost. Second, gzip is dominated on every axis that matters: its 74.90 MiB file is larger than either Zstandard result, and its 44.3629 s median write is about 47 times slower than Zstandard level 3. gzip is the only codec whose write time is measured in tens of seconds rather than fractions of a second.

Third, snappy and lz4 behave almost identically here: 102.17 MiB against 102.65 MiB, with writes of 0.8760 s and 0.8580 s respectively. Both are fast, lightly compressing codecs that shrink the file to about 82 percent of the uncompressed size. snappy is the writer’s default, and neither of these two clearly beats the other on this dataset. It is worth noting that the Parquet specification marks the older LZ4 codec as deprecated in favor of a newer LZ4_RAW framing; the codec exercised here is the one PyArrow writes under the lz4 name.

Read performance tells a flatter story. On a warm page cache, decoding the full table took between 0.0739 s (no compression) and 0.1048 s (gzip). Zstandard sat between the extremes at 0.0783 s for level 3 and 0.0755 s for level 9, and snappy and lz4 landed at 0.0890 s and 0.0858 s. In other words, choosing a strong general-purpose codec added only a few milliseconds to a full decode of the whole table relative to no compression at all, while gzip again stood out as the slowest to read. When the read touched only the single amount column, the absolute times dropped by roughly an order of magnitude, but the same ordering held, with gzip slowest at 0.0352 s.

Filtered reads and the price of decompression

Parquet stores per-row-group statistics, including the minimum and maximum of each column, which lets a reader skip entire row groups whose range cannot satisfy a predicate. This is predicate pushdown, and its benefit depends on the data being sorted so that ranges do not overlap. Because the table was sorted by ts, a predicate of ts >= a late cutoff — matching 498,711 rows, or 4.99 percent of the table — allowed the reader to skip most row groups.

The benchmark measured two versions of that filtered read. The pushdown version passed the predicate to the reader, which used row-group statistics to skip groups. The control version read the relevant columns in full and filtered them in memory afterward, doing the work that pushdown avoids. The contrast is clearest for the compressed codecs. For Zstandard level 3, the pushdown read took 0.0173 s while the no-pushdown control took 0.0406 s; for snappy the two were 0.0170 s and 0.0303 s. Skipping row groups avoids decompressing them, so the saving grows with how expensive the codec is to decode.

The uncompressed case is the instructive exception. With no compression, the pushdown read (0.0176 s) was marginally slower than the no-pushdown control (0.0163 s), because there is no decompression to avoid and the pushdown path carries a small bookkeeping overhead. In short, predicate pushdown and compression reinforce each other: the more a codec costs to decode, the more a query gains by skipping data it never needed to read. This interaction is central to why lakehouse query engines lean heavily on statistics-based skipping over columnar files.

Row-group size: compression against skipping granularity

A row group is the horizontal partition of a Parquet file within which columns are chunked and compressed. Its size controls two things at once. Larger row groups give the codec more context and amortize per-group overhead, which improves compression; smaller row groups create more, finer boundaries, which lets predicate pushdown skip data more precisely. These pull in opposite directions, and the benchmark measured the tension directly by fixing the codec at Zstandard level 3 and varying the row-group size across 131,072 rows, 1,000,000 rows, and 5,000,000 rows.

The compression trend is monotonic: the file shrank from 87.41 MiB at 131,072 rows per group to 68.61 MiB at one million and 66.79 MiB at five million. Small row groups pay a real penalty, here roughly 20 MiB, because encoding structures such as the dictionary reset at every group boundary and the codec sees less data per chunk. The selective-read trend runs the other way. With 77 fine-grained groups, the pushdown filter completed in 0.0070 s, because the reader could discard almost every group; with only two coarse groups it took 0.0853 s, because a matching predicate forces reading a full half of the table. The full-scan read was fastest at the smallest row-group size (0.0587 s) and slowest at the largest (0.1922 s) in this run, though full-scan timing is more sensitive to how decode work parallelizes across groups and should be treated with more caution than the size and pushdown figures.

Dictionary encoding on a low-cardinality column

Dictionary encoding replaces repeated column values with small integer indices into a per-column dictionary, and it is applied before the compression codec runs. For a column with few distinct values it can shrink the data dramatically on its own, which changes how much work is left for the codec. The category column is a natural test: 10 distinct string values repeated across 10,000,000 rows. Holding the codec at Zstandard level 3, the table was written once with dictionary encoding on and once with it off.

The effect was large. With dictionary encoding on, the category column occupied 3.912 MiB compressed on disk, having been reduced to 4.813 MiB uncompressed before the codec even ran. With dictionary encoding off, the same column expanded to 104.825 MiB uncompressed — the raw repeated strings — and although Zstandard still compressed that down, it landed at 14.172 MiB, about 3.6 times the size of the dictionary-encoded column. The whole-file impact followed: the file grew from 68.61 MiB with the dictionary to 87.40 MiB without it, an increase of 18.79 MiB attributable to a single column.

The practical lesson is that for low-cardinality columns, dictionary encoding can matter more than the choice of codec. A strong general-purpose codec applied to raw repeated strings did not recover what dictionary encoding achieved almost for free. PyArrow enables dictionary encoding by default, and the Parquet specification defines the modern RLE_DICTIONARY encoding for exactly this pattern. The failure mode to watch for is a high-cardinality column where the dictionary grows too large to help; the format falls back to plain encoding in that case, and forcing dictionary encoding on such a column wastes effort. The behavior is well matched to the kind of categorical fields common in batch and streaming data pipelines.

Choosing a codec: practical guidance

The measurements support a short, honest set of recommendations for this class of tabular, analytics-oriented data. They are grounded in one dataset on one machine, so they are guidance rather than universal law, but the gaps between codecs are large enough to be robust to modest variation.

Zstandard at a low level is the strongest default. Level 3 produced the smallest practical file (68.61 MiB) at a write cost (0.9421 s) close to the fast codecs and a read cost (0.0783 s) within a few milliseconds of no compression at all. It is the setting that most cleanly balances the three axes. Raising the level to 9 is not worth it on data like this, where it saved under 0.10 MiB for more than double the write time; higher Zstandard levels earn their cost only when the data compresses much further and storage or transfer dominates.

snappy or lz4 make sense when write throughput and decode latency matter more than storage, and the data is written far more often than it is read. Both wrote in about 0.86 to 0.88 s and produced files near 102 MiB, meaning they leave roughly a third of the compressible space on the table in exchange for speed and simplicity. snappy being the PyArrow default is a reasonable choice for intermediate or short-lived files. gzip has no winning case in these results: it was both larger and far slower to write than Zstandard, so Zstandard should be preferred wherever gzip might have been chosen for ratio. Leaving data uncompressed is defensible only for very short-lived scratch files where even a sub-second write matters and the 125.06 MiB footprint is irrelevant.

Two settings beyond the codec deserve equal attention. Row-group size should be tuned to the query pattern: around one million rows is a sound default for time-sorted data read by range, capturing most of the compression benefit while keeping selective reads fast. Dictionary encoding should be left on for low-cardinality columns, where it delivered a larger saving than the codec choice itself. These file-level decisions carry across the broader storage stack, including the open table formats that manage collections of Parquet files, and they compound with engine-level execution choices such as those examined in the article on Apache Spark internals.

Frequently Asked Questions

Which Parquet compression codec is the best default?

On this benchmark, Zstandard at level 3 was the strongest all-round default: it produced the smallest practical file at 68.61 MiB while writing in 0.9421 s and reading in 0.0783 s, within a few milliseconds of no compression. snappy, which PyArrow uses by default, is a reasonable alternative when write speed matters more than storage, but it left the file about 50 percent larger at 102.17 MiB.

Is a higher Zstandard compression level worth the cost?

Not on data like this. Moving from level 3 to level 9 reduced the file by under 0.10 MiB, from 68.61 MiB to 68.52 MiB, while write time rose from 0.9421 s to 2.3837 s, roughly two and a half times longer. Higher levels earn their cost only when the data compresses substantially further and storage or network transfer, rather than write throughput, is the binding constraint.

Why was gzip so much slower than the other codecs?

gzip recorded a median write time of 44.3629 s, about 47 times the 0.9421 s of Zstandard level 3, and it also produced a larger file (74.90 MiB versus 68.61 MiB) and the slowest full read (0.1048 s). In these results gzip was dominated on every axis, so Zstandard is the better choice wherever gzip might have been selected for compression ratio.

How does row-group size affect compression and query speed?

Smaller row groups compress worse but skip data more finely. At 131,072 rows per group the file was 87.41 MiB but a selective pushdown read took only 0.0070 s; at 5,000,000 rows per group the file shrank to 66.79 MiB but the same read rose to 0.0853 s. Around one million rows balanced the two, giving 68.61 MiB and a 0.0176 s selective read.

References

• Benchmark source, methodology, environment capture, and recorded results: jkc4416/analytics-bench (parquet-codecs study).
• Apache Parquet format, compression codecs: parquet.apache.org — Compression.
• Apache Parquet format, dictionary and RLE encodings: parquet.apache.org — Encodings.
• PyArrow writer parameters (compression, compression_level, use_dictionary, row_group_size): arrow.apache.org — pyarrow.parquet.write_table.
• Zstandard compression library and levels: facebook/zstd.

Conclusion

Across one 10,000,000-row table measured with a fixed, recorded methodology, the codec that produced the smallest file was not the one worth choosing for most workloads. Zstandard at level 3 sat at the balanced point: 68.61 MiB on disk, a 0.9421 s write, and a read cost within a few milliseconds of no compression. Level 9 spent more than twice the write time to save under a tenth of a megabyte, and gzip was both larger and dramatically slower, with a 44.3629 s median write. Two file-level settings mattered as much as the codec — a moderate row-group size preserved most of the compression benefit while keeping selective reads fast, and dictionary encoding on a single low-cardinality column changed the whole-file size by 18.79 MiB. The durable conclusion is that Parquet compression is a system of interacting choices, and the right decision comes from measuring the levers in order of impact rather than optimizing any one of them in isolation.