DataFrames at Scale Comparison: TPC-H

May 14, 2024 14m read Hendrik Makait, Sarah Johnson, Matthew Rocklin We run benchmarks derived from the TPC-H benchmark suite on a variety of scales, hardware architectures, and dataframe projects, notably Apache Spark, Dask, DuckDB, and Polars. No project wins. This post analyzes results within each project and between projects.