Uploaded June 2026 | Updated September 2026, 2 weeks ago

---

"Upgrading Apache Spark applications takes 3-6 months per major version. Troubleshooting production failures consumes
20-40% of data engineering time. These two workflows are the biggest time sinks teams face today.

In this session, we'll live-demo two AI-powered agents that compress these workflows from weeks to minutes. The Spark
Troubleshooting Agent diagnoses real production failures — executor OOM, Iceberg schema drift, YARN container kills —
by automatically correlating signals across driver logs, container diagnostics, and Spark event history. The Spark
Upgrade Agent takes a PySpark application from Spark 3.5 to 4.0 on EMR Serverless — iteratively fixing breaking
changes through runtime validation until the job succeeds, with data quality comparison at the end.

Everything happens through natural language. No manual log searching. No reading migration guides. Just describe the
problem and let the agent work."