DuckDB vs Polars for In-Process Analytics: A Practical Comparison

By kongastral

Published June 21, 2026 · Updated July 19, 2026 · 30 min read

On a single 10-million-row Parquet file of about 121 MiB, five analytical queries ran between 1.85 and 5.14 times faster under DuckDB than under Polars, and DuckDB’s peak memory stayed under 130 MiB while Polars reached roughly 1.1 GiB on the heaviest query. Those figures were measured for this comparison on an Apple M2 Pro laptop and are reported in full, with the raw terminal output, further down. They describe one modest, in-memory, single-file workload rather than a universal ranking, and the same two engines trade the lead at larger scales in independent public benchmarks. That tension is the reason this comparison pairs first-hand measurement with the published evidence: DuckDB and Polars are close enough that the right choice depends on scale, workload, memory behavior, and the interface a team prefers, not on a single headline number.

DuckDB and Polars have become two of the most widely adopted tools for analytical work that runs inside a single application process, rather than against a separate database server or a distributed cluster. Both read columnar files such as Parquet directly, both execute queries over an in-memory layout derived from the Apache Arrow format, and both are fast enough that many data teams now reach for one of them before considering a cluster engine such as Apache Spark. They are, however, built around different ideas: DuckDB is a relational database that speaks SQL, while Polars is a DataFrame engine driven by an expression API and written in Rust. This comparison examines their execution models, the memory behavior each shows under load, a reproducible benchmark measured on one machine, the public benchmark evidence at larger scale, and the often-overlooked fact that the two can hand data to each other with no copy.

Two engines, one process: what “in-process analytics” means

The term in-process describes software that runs within the address space of the calling application rather than as a separate service that the application contacts over a network or a socket. A traditional analytical database such as a data warehouse runs as a server: an application opens a connection, sends SQL over the wire, and receives rows back. An in-process engine instead loads as a library inside the program — a Python interpreter, for example — and operates on data in the same memory the program already holds. There is no server to start, no port to manage, and no connection pool to tune.

DuckDB describes itself as “an in-process analytical (OLAP) database.” OLAP, or online analytical processing, refers to read-heavy workloads that scan and aggregate large numbers of rows — the opposite of online transaction processing (OLTP), which favors many small, indexed reads and writes. Polars describes itself as “an analytical query engine written for DataFrames.” A DataFrame is a two-dimensional table abstraction, familiar from libraries such as pandas, in which columns carry typed values and operations are expressed as transformations of whole columns. The two descriptions point at the same workload — analytical queries over tabular data — approached from a relational direction (DuckDB) and a DataFrame direction (Polars). (Sources: duckdb.org/why_duckdb; pypi.org/project/polars, as of 2026-06-22.)

The distinction from a server architecture matters for how a system is built and operated. The figure below contrasts the two deployment shapes.

Neither engine is a distributed system, and this is the single most useful framing for someone deciding when to use them. Apache Spark, the dominant cluster engine for large-scale processing, distributes work across many machines coordinated by a driver and scheduler. That model is appropriate when a dataset genuinely does not fit on one machine or when a workload must be elastic across a fleet. DuckDB and Polars instead exploit a single machine well — every core, the full memory hierarchy, and fast local or cloud storage — and increasingly that is sufficient. Cloud instances now offer hundreds of gigabytes of RAM, and many analytical datasets are smaller than teams assume. When a job fits comfortably on one node, an in-process engine removes the operational weight of a cluster while often finishing faster, because no time is lost to network shuffles or task scheduling.

Because both engines sit at the compute layer, they read from the storage and table-format layer below them rather than replacing it. A team standardizing on open table formats may want to review how those formats differ in the comparison of Apache Iceberg, Delta Lake, and Hudi table formats, since DuckDB and Polars are the engines that would read tables stored in those formats.

Architecture compared: SQL versus DataFrame, eager versus lazy

The two engines share a foundation and diverge above it. Both are columnar, meaning they store and process data column by column rather than row by row. Column-oriented layout suits analytical queries because such queries typically touch a few columns across many rows; reading only the needed columns reduces input/output and lets the processor work on long, uniform runs of values. Both are also vectorized: instead of evaluating one value at a time, they process a batch of values — a vector — in each step of execution, which keeps the processor’s pipeline full and amortizes the overhead of interpreting each operation. DuckDB states that in its engine “queries are still interpreted, but a large batch of values (a ‘vector’) are processed in one operation.” Polars lists “SIMD” operations, referring to Single Instruction, Multiple Data, a processor capability that applies one instruction to several data elements at once. (Sources: duckdb.org/why_duckdb; pypi.org/project/polars, as of 2026-06-22.)

SQL engine versus DataFrame engine

The most visible difference is the interface. DuckDB executes SQL, the declarative query language used by relational databases for five decades. A user writes what result is wanted and the engine decides how to produce it. DuckDB also provides “ACID guarantees through our custom, bulk-optimized Multi-Version Concurrency Control (MVCC)” — ACID being the set of transaction properties (atomicity, consistency, isolation, durability) that keep concurrent changes correct, and MVCC being a technique that lets readers and writers proceed without blocking each other by keeping multiple versions of a row. (Source: duckdb.org/why_duckdb, as of 2026-06-22.)

Polars exposes “a DataFrame expression API plus a SQL interface.” Its primary interface is the expression API, in which transformations are composed as method calls and column expressions in the host language. Polars is “written in Rust,” uses the “Apache Arrow Columnar Format,” and offers frontends in “Python, Rust, NodeJS, R, [and] SQL.” A DataFrame engine appeals to programmers who prefer composing transformations as code, with the full control flow and tooling of a general-purpose language, over embedding SQL strings. (Source: pypi.org/project/polars, as of 2026-06-22.)

Parallelism is realized differently. DuckDB uses “morsel-driven parallelism,” a scheduling approach in which input data is divided into small, fixed-size chunks called morsels that worker threads pull and process, so that load balances across cores without a rigid up-front partitioning. Polars lists “multi-threaded” execution combined with SIMD. Both aim at the same outcome — saturating all available cores on one machine — through different scheduling strategies. (Sources: duckdb.org/why_duckdb; pypi.org/project/polars, as of 2026-06-22.)

Eager versus lazy evaluation

Polars makes an explicit distinction that shapes how performant code is written: eager versus lazy evaluation. In eager mode, each operation runs immediately and returns a materialized result, which is convenient for interactive exploration. In lazy mode, operations build a query plan that is not executed until a terminal call — collect() — is reached. Deferring execution lets the optimizer rewrite the whole plan before any work happens: it can push filters down to the data source so that fewer rows are read, prune unused columns, and reorder operations. Polars documents both modes as “Lazy | Eager execution” with “Query optimization.” The measured benchmark below uses the lazy API for Polars for exactly this reason. (Source: pypi.org/project/polars, as of 2026-06-22.)

DuckDB does not expose an eager/lazy switch because SQL is declarative by nature: a complete statement is handed to the optimizer, which always sees the whole query before execution. The lazy mode of Polars is, in effect, a way to recover that whole-query view inside an imperative DataFrame API. For a data engineer, the practical guidance is that Polars code intended for production should use the lazy API, since eager chains forgo the optimizations that make the engine fast on large inputs.

Streaming and larger-than-memory execution

A defining concern for single-node engines is what happens when a dataset is larger than available RAM. Polars supports processing data “in a streaming fashion,” enabled with collect(engine='streaming'), which executes the query in chunks so that the full dataset need not be resident at once. DuckDB, as a database, has long been designed to spill intermediate state to disk when memory is exhausted, allowing queries to complete on data larger than memory. Both engines therefore offer a path beyond the RAM ceiling, though, as the benchmark section shows, their default memory footprints can differ substantially. The measured benchmark below deliberately uses a dataset that fits in RAM, so it exercises the in-memory path rather than streaming or spilling. (Sources: pypi.org/project/polars; duckdb.org/why_duckdb, as of 2026-06-22.)

Both engines build on the Apache Arrow columnar memory format, an open standard for representing tabular data in memory so that different tools can share buffers without reformatting. Arrow is the reason the two engines interoperate so cheaply, a point developed in the interoperability section. The same Parquet files — Parquet being the dominant open columnar file format on disk — can be read directly by either engine, which means a team is not locked into a proprietary storage format by either choice.

A measured benchmark on one machine

Public benchmarks are useful but abstract; a reproducible measurement on a known machine gives a concrete anchor. The benchmark described in this section was run on a single laptop specifically for this comparison, with every recorded number taken directly from the program’s output. The complete code, the dataset generator, the environment capture, and the raw terminal log are published so the run can be repeated and audited (see the References section). The purpose is not to declare a winner but to show, honestly, how the two engines behaved on one well-defined in-memory workload.

Test environment

All timings were collected on the hardware and software listed below, captured programmatically with sw_vers, sysctl, and the package manager’s list command. Both engines ran with their default thread settings, which resolved to 10 threads on this machine.

Benchmark setup

The dataset was generated by a seeded script (seed 20260719) and written to a single Snappy-compressed Parquet file. Its shape was recorded at generation time:

• Rows: 10,000,000
• File size: 126,976,888 bytes (121.09 MiB)
• Columns: ts (microsecond timestamp over a 90-day window, unsorted), user_id (32-bit integer, Zipf-skewed across 500,000 distinct users), category (one of 10 strings, dictionary-encoded, drawn from a skewed distribution), amount (64-bit float, log-normal, rounded to cents), and flag (boolean, roughly 30 percent true).
• Dimension table: a small dims table of 10 rows (category, region, tier) for the join query.

Five queries were defined with identical intent for both engines. DuckDB ran SQL over read_parquet(...); Polars used scan_parquet(...) lazy frames collected with .collect(). Each query fully materialized its result, so the timing includes producing the complete output, not just planning it.

The measurement protocol was designed to keep the numbers clean. Each engine-query pair ran in a fresh Python subprocess, so the peak-memory reading for one query could not be inflated by memory another query had left resident. Each query ran one warmup pass followed by three timed passes, and the reported time is the median of the three timed passes, measured with a wall-clock timer around the full materialization call. Peak memory is the maximum resident set size (RSS) observed during the timed runs, sampled about every two milliseconds by a background thread and re-anchored at the start of each run. One caveat on the join query is worth stating in advance: in the Parquet file, category is dictionary-encoded and Polars reads it as a categorical type, whereas the dimension table’s key is a plain string, so both keys were cast to string before the join. That cast cost is included in the Polars timing for q4 and is discussed with the results.

Query time results

On this dataset DuckDB was faster on every query. The ratio in the final column is the Polars median divided by the DuckDB median, so a value of 3.05 means DuckDB completed that query about three times faster.

The gap ranged from a factor of 1.85 on the simple full-scan count to a factor of 5.14 on the join. The join result deserves the caveat noted earlier: on the Polars side the join key was cast from a categorical type to a string before the operation, and that cast is counted in the 0.1763-second figure. Part of the 5.14 ratio therefore reflects a data-typing detail rather than pure engine speed, and a schema that avoided the cast would likely narrow it. The smallest gap, on scan_count_distinct, is consistent with a query dominated by reading and hashing a single high-cardinality column, where both engines spend most of their time in similar work.

Peak memory results

The memory picture was more lopsided than the timing picture. DuckDB’s peak resident memory stayed between 77 and 123 MiB across the five queries, close to the on-disk size of the data. Polars, running its default in-memory collection, ranged from 182 MiB to 1,086 MiB, with the two grouping-heavy queries — time_bucket and join_dims — the most memory-intensive.

The raw terminal output below is copied verbatim from the run log, so the numbers in the tables and charts can be checked against their source.

==============================================================================
RESULTS  (median of 3 timed runs)
==============================================================================
query                 DuckDB med(s)  Polars med(s)   ratio P/D
--------------------------------------------------------------
filtered_groupby             0.0168         0.0511        3.05
highcard_topn                0.0285         0.0940        3.30
time_bucket                  0.0396         0.1491        3.76
join_dims                    0.0343         0.1763        5.14
scan_count_distinct          0.0210         0.0388        1.85

PEAK RSS  (max over timed runs, MiB)
query                    DuckDB MiB     Polars MiB
--------------------------------------------------
filtered_groupby               77.0          354.5
highcard_topn                 123.2          463.6
time_bucket                   120.9         1086.1
join_dims                      77.1          886.9
scan_count_distinct          107.0          181.8

ARROW INTEROP  (duckdb.arrow -> polars.from_arrow, median of 3)
  rows_transferred     = 3,001,101
  duckdb -> arrow  (s) = 0.0018
  arrow  -> polars (s) = 0.2055
  round trip total (s) = 0.2073

What the measurements show — and what they do not

Read within its boundaries, this run gives a clear result: for a 10-million-row file that fits in RAM, default DuckDB was faster on all five queries and used far less memory than default Polars. The memory finding is the more decisive of the two, since a query that peaks near 1 GiB rather than near 120 MiB changes how many concurrent jobs a machine can host, even when both finish in a fraction of a second.

The result also corrects an impression that a documentation-only reading of the two engines can leave. Because the Polars project’s own published benchmark shows its streaming engine winning at a 10 GB scale factor, it is tempting to conclude that Polars is the faster choice at small scale in general. This measured workload is smaller still — about 0.12 GB — and default in-memory Polars was slower on every query. The lesson is that a benchmark result belongs to a specific configuration: the Polars streaming figure reflects a particular engine mode and query suite, not a universal small-scale advantage, and the default lazy-then-collect path measured here behaves differently. Any claim that one engine is simply “faster at small data” should be treated with suspicion until the configuration is named.

Several limits keep this from being a verdict. The dataset is small and fits entirely in memory, so it never exercises streaming or disk spilling, where the balance can shift. Polars ran in its default collection mode rather than engine='streaming', and its higher memory reflects that default; a streaming collect would trade some speed for a lower footprint. The join ratio is partly a data-type artifact, as noted. The sample is three timed runs on one laptop with warm file-system cache. And a single-file layout favors neither engine’s partition handling. The independent public benchmarks in the next section, run at 10 GB and 100 GB, show the two engines much closer and the lead changing hands, which is the necessary counterweight to a small in-memory test.

Larger-scale context: PDS-H and a memory stress test

Two public benchmarks extend the picture to scales a laptop test cannot reach, and both must be read with their conditions in mind. Neither is a vendor-neutral, audited TPC-H comparison of the two engines, because none existed as of writing, so they are context rather than a final ranking.

The PDS-H throughput benchmark

The Polars project publishes a benchmark based on PDS-H, a workload derived from TPC-H. TPC-H is a long-standing industry decision-support benchmark consisting of analytical queries over a synthetic dataset whose size is set by a “scale factor” (SF); SF-10 is roughly ten gigabytes of raw data and SF-100 roughly one hundred. The PDS-H total execution times reported by the Polars project (May 2025) are shown below.

Two patterns stand out. First, both DuckDB and Polars are an order of magnitude faster than pandas, PySpark, and Dask across these scales, which supports the broader claim that an in-process columnar engine outperforms both a single-threaded DataFrame library and a cluster engine on a single node at these sizes. Second, the lead changes with scale: Polars streaming was fastest at SF-10 (3.89 s versus DuckDB’s 5.87 s), while DuckDB was fastest at SF-100 (19.65 s versus Polars streaming’s 23.94 s). The collapse of Polars in-memory from 9.68 s at SF-10 to 152.27 s at SF-100 also shows why the streaming engine matters as data grows — and why the in-memory mode measured on the small laptop file is not the mode to use at scale. (Source: pola.rs/posts/benchmarks, as of 2026-06-22.)

The Parquet memory stress test

Peak memory determines whether a job fits on a given machine at all. An independent benchmark published on the codecentric blog (published 2026-01-20, updated 2026-02-02) scaled a single Parquet file from roughly 2 GB to 140 GB and measured both execution time and peak memory. On execution time the two engines were very similar across the range, with DuckDB roughly one second faster at the largest scale. The revealing result was peak memory at 140 GB: DuckDB held roughly 1.3 GB by default while default Polars used roughly 17 GB on the same file, yet forcing asynchronous reads brought Polars down to roughly 750 MB, below DuckDB. Separately, partitioning the 140 GB dataset into 72 smaller files cut DuckDB’s memory by roughly eight times and default Polars’ by roughly four times, showing that file layout strongly influences memory regardless of engine. (Source: codecentric.de blog, as of 2026-06-22.)

The stress test and the laptop measurement agree on direction: DuckDB is conservative with memory out of the box, and default Polars is not. They differ on whether that gap is intrinsic — the stress test shows Polars can be tuned below DuckDB with asynchronous reads at 140 GB, a lever the small in-memory test did not pull. Together they support the same discipline: measure under conditions that resemble the real workload, since the same “measure under your own conditions” principle applies when selecting data stores more broadly, as the comparison of databases for preprocessed time-series data walks through for a related class of workload.

Interoperability: zero-copy handoff over Apache Arrow

A point often missing from “DuckDB versus Polars” discussions is that the two are not mutually exclusive within one program. Because both represent data in the Apache Arrow columnar format, they can share the same in-memory buffers with little or no copying. The official DuckDB documentation states that “DuckDB can read Polars DataFrames and convert query results to Polars DataFrames. It does this internally using the efficient Apache Arrow integration.” In practice, a Polars DataFrame held in a variable can be referenced directly by name inside a SQL query — duckdb.sql("SELECT * FROM df") — and the result can be converted back to a Polars DataFrame with .pl(), or to a Polars LazyFrame with .pl(lazy=True). The pyarrow package is required for this path. (Source: duckdb.org/docs/current/guides/python/polars.html, as of 2026-06-22.)

Zero-copy means the handoff transfers ownership of, or a reference to, the existing memory buffer rather than serializing and reallocating the data. The figure below shows a single Arrow buffer being read by both a Polars DataFrame and a DuckDB query.

The benchmark measured this handoff as well. A filtered result of 3,001,101 rows was moved from DuckDB into an Arrow table and then into a Polars DataFrame. Exporting from DuckDB to Arrow took a median of 0.0018 seconds — effectively free, consistent with a reference handoff rather than a copy. Reconstructing a Polars DataFrame from that Arrow table (polars.from_arrow) took a median of 0.2055 seconds, so the full round trip was 0.2073 seconds for just over three million rows. The practical reading is that the boundary between the two engines is cheap relative to re-reading the data or serializing it across a process, but “zero-copy” does not mean literally zero time; the Polars ingestion step still does measurable work. Even so, moving three million rows in about a fifth of a second is inexpensive next to the cost of the analytical queries themselves at larger scales.

This interoperability changes the decision calculus. A pipeline can perform DataFrame-style feature preparation in Polars, hand the frame to DuckDB for a complex multi-table SQL join and aggregation, and receive the result back as a Polars frame for the next step — all within one process and without paying a full serialization cost at each boundary. The two engines become complementary stages rather than competing choices.

Both engines also fit cleanly into the wider data ecosystem. They read Parquet from local disk and from object storage such as Amazon S3, which is the common substrate for analytical data lakes. DuckDB has an adapter, dbt-duckdb, that lets it serve as the execution engine for transformation models; teams already using that framework can read about the model-based workflow in the guide to building transformation pipelines with dbt. An in-process engine is also a natural transform step inside a scheduled workflow, where an orchestrator triggers the job; the patterns for that are covered in the guide to orchestrating data pipelines with Apache Airflow. When the analytical layer is fed by streaming ingestion, the upstream side is described in the guide to change data capture with Debezium and Kafka.

Choosing: when DuckDB, when Polars, when both

Because the engines trade the performance lead by scale and workload, the most reliable basis for choosing is fit — to the team’s skills, the surrounding system, and the shape of the work — rather than a single benchmark number. The measured laptop test is a reminder that on small in-memory data DuckDB is fast and frugal by default, while the public benchmarks are a reminder that the picture narrows and can reverse at scale. The decision flow below summarizes the practical signals.

The table below condenses the same guidance into a side-by-side reference.

A reasonable default is to choose by interface and team skills first: a team comfortable in SQL, or one that wants transactional guarantees and conservative memory out of the box, will find DuckDB the lower-friction option, while a team writing DataFrame-style transformation code, preparing features for machine learning, or working in Rust will find Polars more natural. When a single pipeline genuinely contains both shapes of work, the low-cost Arrow bridge makes “use both” a sound engineering choice rather than a compromise. The one case where neither is the right answer is data that cannot be made to fit on a single large machine even with streaming or spilling; that workload still belongs on a distributed engine.

Frequently Asked Questions

Which was faster in the measured benchmark, DuckDB or Polars?

On the 10-million-row, 121 MiB Parquet file used here, DuckDB was faster on all five queries — by 1.85 times on a full-scan count and up to 5.14 times on the join — and used far less peak memory (77 to 123 MiB, versus 182 to 1,086 MiB for default Polars). Two qualifications matter: the file fits entirely in RAM, so streaming and disk spilling were never exercised, and part of the largest ratio comes from a categorical-to-string cast on the Polars join key. At 10 GB and 100 GB in the Polars-published PDS-H benchmark, the two engines are much closer and the lead changes hands. Treat the laptop result as one honest data point about a small in-memory workload, not a general ranking. (Measured 2026-07-19; code and raw results linked in the References.)

Can DuckDB and Polars be used together?

Yes. Both represent data in the Apache Arrow columnar format, so they exchange data with little copying. A Polars DataFrame in scope can be queried directly by variable name in SQL, for example duckdb.sql("SELECT * FROM df"), and a DuckDB result converts back to a Polars DataFrame with .pl() or to a LazyFrame with .pl(lazy=True). In the benchmark here, moving a 3,001,101-row result from DuckDB into Arrow took about 0.0018 seconds, and the full round trip into a Polars DataFrame took about 0.2073 seconds. The pyarrow package is required. (Source: duckdb.org/docs/current/guides/python/polars.html, as of 2026-06-22.)

Does the benchmark mean DuckDB is the better choice overall?

No. It means DuckDB was faster and more memory-frugal on one small in-memory, single-file workload with Polars in its default collection mode. The result does not cover larger-than-memory data, streaming collection, partitioned files, or schemas that avoid the join-key cast, and independent large-scale benchmarks show the engines much closer. The durable basis for choosing is interface preference (SQL versus DataFrame), default memory behavior, transactional needs, and ecosystem fit, with the option to use both engines together over Arrow when a pipeline contains both styles of work.

References

• Benchmark code and raw results (dataset generator, benchmark orchestrator, environment capture, and full terminal log for the measured DuckDB vs Polars run reported here), measured 2026-07-19.
• DuckDB — Why DuckDB (in-process OLAP, columnar-vectorized execution, MVCC, morsel-driven parallelism), as of 2026-06-22.
• Polars — PyPI project page (analytical query engine for DataFrames, Apache Arrow format, lazy/eager execution, SIMD, streaming), as of 2026-06-22.
• Polars — PDS-H benchmark (TPC-H-derived, May 2025; results not comparable to certified TPC-H), as of 2026-06-22.
• DuckDB documentation — Integration with Polars (zero-copy Apache Arrow handoff, .pl() conversion), as of 2026-06-22.

Conclusion

DuckDB and Polars represent two routes to the same destination: fast analytical computation inside a single process, over open columnar data, on one machine. DuckDB approaches the problem as a relational database — SQL, ACID transactions through MVCC, conservative default memory, and morsel-driven parallelism — while Polars approaches it as a Rust DataFrame engine with an expressive expression API, explicit lazy evaluation, and SIMD-accelerated multi-threading. The benchmark measured here shows that on a small in-memory file DuckDB was faster on every query and much more frugal with memory in its default mode, while the independent public benchmarks show the two engines close and the lead changing hands at 10 GB and 100 GB. Both readings are true within their conditions, which is precisely why a single number should not decide the matter. Because both build on Apache Arrow, the most consequential and least discussed fact is that they interoperate at low cost, turning an apparent rivalry into a pairing. A sound selection therefore rests on interface preference, memory behavior, and ecosystem fit — and on measuring under conditions that resemble the intended workload — with the option, where a pipeline contains both relational and DataFrame work, to use both engines bridged over Arrow.