How AI is applied across API Evangelist and APIs.io. Read my AI disclosure →
API Evangelist API Evangelist
Discovery
Learnings
Guidance
Toolbox
Alignment
API Evangelist LLC

Optimizing PySpark UDFs on AWS Glue 6.0 with native Arrow UDFs

calendar_today November 13, 2026 person Iván Fernández Perea domain aws-glue

Python UDFs in PySpark can be slow because data is serialized between the JVM and the Python runtime. We benchmark five approaches on AWS Glue 6.0 (standard Python, Arrow-optimized Python, vectorized pandas, native Arrow UDFs, and Spark SQL) across 10M to 100M rows, and separate transport cost from compute cost to explain the ranking.

open_in_new Read original post