Using LLMs to Analyze Spark SQL Plans: A Practical Approach to Debugging Long-Running Jobs

Expedia Group Technology — Innovation Using large language models to reveal bottlenecks in Spark SQL execution plans Photo by Luis del Río If you’ve ever stared at a 300-plus-node physical plan at 2 a.m. trying to spot a missing broadcast or one cursed skewed partition, this is for you. Spark makes it deceptively easy to write complex SQL that looks correct but quietly turns into a performance and cost problem at scale.