Improve performance for Python UDFs and complex data types in Microsoft Fabric’s Native Execution Engine

Where this matters: A concrete example For example, you have a data pipeline (extract, transform, load—ETL) that ingests JSON telemetry events, each containing nested arrays of user actions. Your pipeline uses a Python UDF to apply custom business rules—say, classifying events based on logic that doesn’t map cleanly to SQL expressions. Before NEE, this pipeline hits two performance penalties at once: The Python/Scala UDF forces serialization between the JVM and a Python worker for every batch.