DuckDB vs Polars for In-Process Analytics: A Practical Comparison

DuckDB and Polars have become two of the most widely adopted tools for analytical work that runs inside a single application process, rather than against a separate database server or a distributed cluster. Both read columnar files such as Parquet directly, both execute queries over an in-memory layout derived from the Apache Arrow format, and both are fast enough that many data teams now reach for one of them before considering a cluster engine such as Apache Spark. They are, however, built around different ideas: DuckDB is a relational database that speaks SQL, while Polars is a DataFrame engine driven by an expression API and written in Rust. This post compares the two as engineering tools — their execution models, their memory behavior under load, the benchmark evidence that exists (and its limits), and the often-overlooked fact that they can hand data to each other with no copy. The goal is not to crown a single winner, because the two trade the performance lead depending on scale and workload, but to give a data engineer a clear basis for choosing one, the other, or both.

Two engines, one process: what “in-process analytics” means

The term in-process describes software that runs within the address space of the calling application rather than as a separate service that the application contacts over a network or a socket. A traditional analytical database such as a data warehouse runs as a server: an application opens a connection, sends SQL over the wire, and receives rows back. An in-process engine instead loads as a library inside the program — a Python interpreter, for example — and operates on data in the same memory the program already holds. There is no server to start, no port to manage, and no connection pool to tune.

DuckDB describes itself as “an in-process analytical (OLAP) database.” OLAP, or online analytical processing, refers to read-heavy workloads that scan and aggregate large numbers of rows — the opposite of online transaction processing (OLTP), which favors many small, indexed reads and writes. Polars describes itself as “an analytical query engine written for DataFrames.” A DataFrame is a two-dimensional table abstraction, familiar from libraries such as pandas, in which columns carry typed values and operations are expressed as transformations of whole columns. The two descriptions point at the same workload — analytical queries over tabular data — approached from a relational direction (DuckDB) and a DataFrame direction (Polars). (Sources: duckdb.org/why_duckdb; pypi.org/project/polars, as of 2026-06-22.)

The distinction from a server architecture matters for how a system is built and operated. The figure below contrasts the two deployment shapes.

Neither engine is a distributed system, and this is the single most useful framing for someone deciding when to use them. Apache Spark, the dominant cluster engine for large-scale processing, distributes work across many machines coordinated by a driver and scheduler. That model is appropriate when a dataset genuinely does not fit on one machine or when a workload must be elastic across a fleet. DuckDB and Polars instead exploit a single machine well — every core, the full memory hierarchy, and fast local or cloud storage — and increasingly that is sufficient. Cloud instances now offer hundreds of gigabytes of RAM, and many analytical datasets are smaller than teams assume. When a job fits comfortably on one node, an in-process engine removes the operational weight of a cluster while often finishing faster, because no time is lost to network shuffles or task scheduling.

Because both engines sit at the compute layer, they read from the storage and table-format layer below them rather than replacing it. A team standardizing on open table formats may want to review how those formats differ in the comparison of Apache Iceberg, Delta Lake, and Hudi table formats, since DuckDB and Polars are the engines that would read tables stored in those formats.

Architecture compared: SQL versus DataFrame, eager versus lazy

The two engines share a foundation and diverge above it. Both are columnar, meaning they store and process data column by column rather than row by row. Column-oriented layout suits analytical queries because such queries typically touch a few columns across many rows; reading only the needed columns reduces input/output and lets the processor work on long, uniform runs of values. Both are also vectorized: instead of evaluating one value at a time, they process a batch of values — a vector — in each step of execution, which keeps the processor’s pipeline full and amortizes the overhead of interpreting each operation. DuckDB states that in its engine “queries are still interpreted, but a large batch of values (a ‘vector’) are processed in one operation.” Polars lists “SIMD” operations, referring to Single Instruction, Multiple Data, a processor capability that applies one instruction to several data elements at once. (Sources: duckdb.org/why_duckdb; pypi.org/project/polars, as of 2026-06-22.)

SQL engine versus DataFrame engine

The most visible difference is the interface. DuckDB executes SQL, the declarative query language used by relational databases for five decades. A user writes what result is wanted and the engine decides how to produce it. DuckDB also provides “ACID guarantees through our custom, bulk-optimized Multi-Version Concurrency Control (MVCC)” — ACID being the set of transaction properties (atomicity, consistency, isolation, durability) that keep concurrent changes correct, and MVCC being a technique that lets readers and writers proceed without blocking each other by keeping multiple versions of a row. (Source: duckdb.org/why_duckdb, as of 2026-06-22.)

Polars exposes “a DataFrame expression API plus a SQL interface.” Its primary interface is the expression API, in which transformations are composed as method calls and column expressions in the host language. Polars is “written in Rust,” uses the “Apache Arrow Columnar Format,” and offers frontends in “Python, Rust, NodeJS, R, [and] SQL.” A DataFrame engine appeals to programmers who prefer composing transformations as code, with the full control flow and tooling of a general-purpose language, over embedding SQL strings. (Source: pypi.org/project/polars, as of 2026-06-22.)

Parallelism is realized differently. DuckDB uses “morsel-driven parallelism,” a scheduling approach in which input data is divided into small, fixed-size chunks called morsels that worker threads pull and process, so that load balances across cores without a rigid up-front partitioning. Polars lists “multi-threaded” execution combined with SIMD. Both aim at the same outcome — saturating all available cores on one machine — through different scheduling strategies. (Sources: duckdb.org/why_duckdb; pypi.org/project/polars, as of 2026-06-22.)

Eager versus lazy evaluation

Polars makes an explicit distinction that shapes how performant code is written: eager versus lazy evaluation. In eager mode, each operation runs immediately and returns a materialized result, which is convenient for interactive exploration. In lazy mode, operations build a query plan that is not executed until a terminal call — collect() — is reached. Deferring execution lets the optimizer rewrite the whole plan before any work happens: it can push filters down to the data source so that fewer rows are read, prune unused columns, and reorder operations. Polars documents both modes as “Lazy | Eager execution” with “Query optimization.” (Source: pypi.org/project/polars, as of 2026-06-22.)

DuckDB does not expose an eager/lazy switch because SQL is declarative by nature: a complete statement is handed to the optimizer, which always sees the whole query before execution. The lazy mode of Polars is, in effect, a way to recover that whole-query view inside an imperative DataFrame API. For a data engineer, the practical guidance is that Polars code intended for production should use the lazy API, since eager chains forgo the optimizations that make the engine fast on large inputs.

Streaming and larger-than-memory execution

A defining concern for single-node engines is what happens when a dataset is larger than available RAM. Polars supports processing data “in a streaming fashion,” enabled with collect(engine='streaming'), which executes the query in chunks so that the full dataset need not be resident at once. DuckDB, as a database, has long been designed to spill intermediate state to disk when memory is exhausted, allowing queries to complete on data larger than memory. Both engines therefore offer a path beyond the RAM ceiling, though, as the benchmark section shows, their default memory footprints under stress can differ substantially. (Sources: pypi.org/project/polars; duckdb.org/why_duckdb, as of 2026-06-22.)

Both engines build on the Apache Arrow columnar memory format, an open standard for representing tabular data in memory so that different tools can share buffers without reformatting. Arrow is the reason the two engines interoperate so cheaply, a point developed in the interoperability section. The same Parquet files — Parquet being the dominant open columnar file format on disk — can be read directly by either engine, which means a team is not locked into a proprietary storage format by either choice.

Performance and memory: what the benchmarks actually show

Performance comparisons between DuckDB and Polars are widely circulated and widely misread. Two public benchmarks are credible enough to discuss, and both must be read with their conditions in mind. The first is a throughput benchmark published by the Polars project; the second is a memory-focused stress test from an independent consultancy. No vendor-neutral, audited TPC-H comparison of the two engines existed as of writing, so these are the most credible public figures available, and neither should be treated as the final word.

The PDS-H throughput benchmark

The Polars project publishes a benchmark based on PDS-H, a workload derived from TPC-H. TPC-H is a long-standing industry decision-support benchmark consisting of analytical queries over a synthetic dataset whose size is set by a “scale factor” (SF); SF-10 is roughly ten gigabytes of raw data and SF-100 roughly one hundred. The PDS-H total execution times reported by the Polars project (May 2025) are shown below.

Two patterns stand out. First, both DuckDB and Polars are an order of magnitude faster than pandas, PySpark, and Dask across these scales, which supports the broader claim that an in-process columnar engine outperforms both a single-threaded DataFrame library and a cluster engine on a single node at these sizes. Second, the lead changes with scale: Polars streaming was fastest at SF-10 (3.89 s versus DuckDB’s 5.87 s), while DuckDB was fastest at SF-100 (19.65 s versus Polars streaming’s 23.94 s). The collapse of Polars in-memory from 9.68 s at SF-10 to 152.27 s at SF-100 also shows why the streaming engine matters as data grows. (Source: pola.rs/posts/benchmarks, as of 2026-06-22.)

The Parquet memory stress test

Throughput is only one axis; peak memory determines whether a job fits on a given machine at all. An independent benchmark published on the codecentric blog (published 2026-01-20, updated 2026-02-02) scaled a single Parquet file from roughly 2 GB to 140 GB and measured both execution time and peak memory. On execution time the two engines were very similar across the range, with DuckDB roughly one second faster at the largest scale. The more revealing result was peak memory at 140 GB, summarized below.

Two findings deserve emphasis. First, default behavior diverged by more than an order of magnitude: DuckDB held about 1.3 GB while default Polars used about 17 GB on the same 140 GB file. Second, the gap was not intrinsic — forcing asynchronous reads brought Polars down to about 750 MB, below DuckDB’s footprint. Separately, partitioning the 140 GB dataset into 72 smaller files cut DuckDB’s memory by roughly eight times and default Polars’ by roughly four times, showing that file layout strongly influences memory regardless of engine. (Source: codecentric.de blog, as of 2026-06-22.)

The combined reading of the two benchmarks is consistent with the framing that neither engine is universally faster. On throughput the lead trades by scale; on memory DuckDB is more conservative out of the box, but Polars can match or beat it once tuned, and file layout dominates both. For a team selecting between data stores and engines more broadly, the same “measure under your own conditions” discipline applies; the comparison of databases for preprocessed time-series data walks through a similar evaluation for a related class of workload.

Interoperability: zero-copy handoff over Apache Arrow

A point often missing from “DuckDB versus Polars” discussions is that the two are not mutually exclusive within one program. Because both represent data in the Apache Arrow columnar format, they can share the same in-memory buffers without copying. The official DuckDB documentation states that “DuckDB can read Polars DataFrames and convert query results to Polars DataFrames. It does this internally using the efficient Apache Arrow integration.” In practice, a Polars DataFrame held in a variable can be referenced directly by name inside a SQL query — duckdb.sql("SELECT * FROM df") — and the result can be converted back to a Polars DataFrame with .pl(), or to a Polars LazyFrame with .pl(lazy=True). The pyarrow package is required for this path. (Source: duckdb.org/docs/current/guides/python/polars.html, as of 2026-06-22.)

Zero-copy means the handoff transfers ownership of, or a reference to, the existing memory buffer rather than serializing and reallocating the data. The figure below shows a single Arrow buffer being read by both a Polars DataFrame and a DuckDB query.

This interoperability changes the decision calculus. A pipeline can perform DataFrame-style feature preparation in Polars, hand the frame to DuckDB for a complex multi-table SQL join and aggregation, and receive the result back as a Polars frame for the next step — all within one process and without paying a serialization cost at each boundary. The two engines become complementary stages rather than competing choices.

Both engines also fit cleanly into the wider data ecosystem. They read Parquet from local disk and from object storage such as Amazon S3, which is the common substrate for analytical data lakes. DuckDB has an adapter, dbt-duckdb, that lets it serve as the execution engine for transformation models; teams already using that framework can read about the model-based workflow in the guide to building transformation pipelines with dbt. An in-process engine is also a natural transform step inside a scheduled workflow, where an orchestrator triggers the job; the patterns for that are covered in the guide to orchestrating data pipelines with Apache Airflow. When the analytical layer is fed by streaming ingestion, the upstream side is described in the guide to change data capture with Debezium and Kafka.

Choosing: when DuckDB, when Polars, when both

Because the engines trade the performance lead by scale and workload, the most reliable basis for choosing is fit — to the team’s skills, the surrounding system, and the shape of the work — rather than a single benchmark number. The decision flow below summarizes the practical signals.

The table below condenses the same guidance into a side-by-side reference.

A reasonable default is to choose by interface and team skills first: a team comfortable in SQL, or one that wants transactional guarantees and conservative memory out of the box, will find DuckDB the lower-friction option, while a team writing DataFrame-style transformation code, preparing features for machine learning, or working in Rust will find Polars more natural. When a single pipeline genuinely contains both shapes of work, the zero-copy Arrow bridge makes “use both” a sound engineering choice rather than a compromise. The one case where neither is the right answer is data that cannot be made to fit on a single large machine even with streaming or spilling; that workload still belongs on a distributed engine.

Frequently Asked Questions

Is DuckDB or Polars faster?

Neither is faster in all cases. In the Polars-published PDS-H benchmark (May 2025), Polars streaming was fastest at scale factor 10 (3.89 s versus DuckDB’s 5.87 s) while DuckDB was fastest at scale factor 100 (19.65 s versus 23.94 s). In the independent codecentric Parquet stress test, execution times were very similar across scales, with DuckDB about one second faster at 140 GB. The lead trades by scale and workload, and both benchmarks carry caveats: the PDS-H figures are vendor-published and not comparable to certified TPC-H results, and the stress test is from a consultancy blog. (Sources: pola.rs/posts/benchmarks; codecentric.de blog, as of 2026-06-22.)

Can DuckDB and Polars be used together?

Yes. Both represent data in the Apache Arrow columnar format, so they exchange data zero-copy. A Polars DataFrame in scope can be queried directly by variable name in SQL, for example duckdb.sql("SELECT * FROM df"), and a DuckDB result converts back to a Polars DataFrame with .pl() or to a LazyFrame with .pl(lazy=True). The pyarrow package is required. (Source: duckdb.org/docs/current/guides/python/polars.html, as of 2026-06-22.)

What does “in-process” mean, and how is it different from Spark?

In-process means the engine runs as a library inside the application’s own process rather than as a separate server. DuckDB and Polars both run on a single machine and read files directly, so they remove the operational overhead of a server or a cluster. Apache Spark is a distributed engine that coordinates work across many machines; it is appropriate when data does not fit on one node or when elasticity across a fleet is required. For workloads that fit on one machine, an in-process engine is often simpler and faster.

Why does Polars distinguish eager and lazy evaluation?

In eager mode each operation runs immediately, which suits interactive exploration. In lazy mode operations build a query plan that is executed only at a collect() call, which lets the optimizer rewrite the whole plan first — pushing filters down, pruning unused columns, and reordering steps. Production Polars code generally uses the lazy API to gain these optimizations. DuckDB has no equivalent switch because SQL is declarative and the optimizer always sees the complete query. (Source: pypi.org/project/polars, as of 2026-06-22.)

Which uses less memory?

In the codecentric stress test on a 140 GB single Parquet file, DuckDB used about 1.3 GB by default while default Polars used about 17 GB, but forcing asynchronous reads brought Polars down to about 750 MB. Partitioning the dataset into 72 files cut DuckDB’s memory roughly eightfold and default Polars’ roughly fourfold. DuckDB is more conservative out of the box, but Polars is tunable, and file layout influences memory strongly for both. (Source: codecentric.de blog, as of 2026-06-22.)

References

• DuckDB — Why DuckDB (in-process OLAP, columnar-vectorized execution, MVCC, morsel-driven parallelism), as of 2026-06-22.
• Polars — PyPI project page (analytical query engine for DataFrames, Apache Arrow format, lazy/eager execution, SIMD, streaming), as of 2026-06-22.
• Polars — PDS-H benchmark (TPC-H-derived, May 2025; results not comparable to certified TPC-H), as of 2026-06-22.
• DuckDB documentation — Integration with Polars (zero-copy Apache Arrow handoff, .pl() conversion), as of 2026-06-22.
• DuckDB 1.5.4 release announcement (current release and 1.4.5 LTS line), as of 2026-06-22.

Conclusion

DuckDB and Polars represent two routes to the same destination: fast analytical computation inside a single process, over open columnar data, on one machine. DuckDB approaches the problem as a relational database — SQL, ACID transactions through MVCC, conservative default memory, and morsel-driven parallelism — while Polars approaches it as a Rust DataFrame engine with an expressive expression API, explicit lazy evaluation, and SIMD-accelerated multi-threading. The public benchmarks confirm that neither holds a universal performance lead; the Polars-published PDS-H figures show the lead changing between scale factor 10 and 100, and the independent codecentric stress test shows DuckDB conservative on memory by default while Polars matches it once tuned, with file layout dominating both. Because both build on Apache Arrow, the most consequential and least discussed fact is that they interoperate zero-copy, which turns an apparent rivalry into a pairing. A sound selection therefore rests on interface preference, memory behavior, and ecosystem fit rather than on a single throughput number — and where a pipeline contains both relational and DataFrame work, the right answer is often to use both, bridged over Arrow.