Uploaded April 2026 | Updated September 2026, 2 weeks ago
Chang She, co-founder and CEO of LanceDB, sits down with Pete Soderling ahead of AI Council SF 2026 to explain why Parquet โ the columnar format that's powered data engineering for the last decade โ couldn't handle the AI workloads his team was building.
After six months of trying to make Spark on Parquet work for large-scale autonomous vehicle data mining, Chang and his team hit two walls: random access performance (it took tens of seconds to fetch just 10-100 rows) and multimodal data storage (keeping raw data, feature data, and analytical data in sync across three systems was unsustainable in production).
In this clip, Chang shares:
- The two challenges that broke Parquet for AI workloads
- Why "physical AI today basically has the same problems"
- What he learned from interviewing 100+ ML and computer vision engineers
- Why modifying Parquet would have meant making it "no longer Parquet"
- The architectural decisions that led to building LanceDB from scratch
Catch Chang and the rest of the speaker lineup at AI Council SF, May 12โ14, 2026 in SOMA: aicouncil.com/sf-2026
#LanceDB #Parquet #AIInfrastructure #DataEngineering #AICouncil
๐ Sign up for our "No BS" Newsletter to get the latest technical data & AI content: aicouncil.com/newsletter
ABOUT AI COUNCIL:
AI Council brings together the brightest minds in data to share industry knowledge, technical architectures and best practices in building cutting edge data & AI systems and tools.
FIND US:
Twitter: twitter.com/AICouncilConf
LinkedIn: linkedin.com/company/datacouncil-ai
Website: datacouncil.ai
Chang She, co-founder and CEO of LanceDB, sits down with Pete Soderling ahead of AI Council SF 2026 to explain why Parquet โ the columnar format that's powered data engineering for the last decade โ couldn't handle the AI workloads his team was building.
After six months of trying to make Spark on Parquet work for large-scale autonomous vehicle data mining, Chang and his team hit two walls: random access performance (it took tens of seconds to fetch just 10-100 rows) and multimodal data storage (keeping raw data, feature data, and analytical data in sync across three systems was unsustainable in production).
In this clip, Chang shares:
- The two challenges that broke Parquet for AI workloads
- Why "physical AI today basically has the same problems"
- What he learned from interviewing 100+ ML and computer vision engineers
- Why modifying Parquet would have meant making it "no longer Parquet"
- The architectural decisions that led to building LanceDB from scratch
Catch Chang and the rest of the speaker lineup at AI Council SF, May 12โ14, 2026 in SOMA: aicouncil.com/sf-2026
#LanceDB #Parquet #AIInfrastructure #DataEngineering #AICouncil
๐ Sign up for our "No BS" Newsletter to get the latest technical data & AI content: aicouncil.com/newsletter
ABOUT AI COUNCIL:
AI Council brings together the brightest minds in data to share industry knowledge, technical architectures and best practices in building cutting edge data & AI systems and tools.
FIND US:
Twitter: twitter.com/AICouncilConf
LinkedIn: linkedin.com/company/datacouncil-ai
Website: datacouncil.ai
