Five years of OpenLineage: How we built an industry standard and why agents need it | Datadog @aicouncilconf
Five years of OpenLineage: How we built an industry standard and why agents need it | Datadog  @aicouncilconf
Uploaded June 2026 | Updated November 2026, 4 days ago
[2026 - DAY 3 - LIGHTNING TALK] Over the past five years, OpenLineage has become the de facto standard for data lineage metadata, adopted across the industry by leading platforms and enterprises. In this talk, we'll trace the journey of building an open standard. You'll learn what changed in the ecosystem that made standardization possible, the critical features that drove adoption (column-level lineage, streaming support, unified facets), and where OpenLineage stands today - five years since its initial release. Most importantly, we'll explore why this matters now: as AI agents increasingly make decisions about data - where to read from, what to trust, how fresh it is - they need a shared understanding of data context. Lineage metadata is the knowledge graph that transforms agents from black boxes into informed decision-makers. The talk covers the standards perspective, the pragmatic integration challenges, and a forward-looking vision for how great metadata enables intelligent data systems.

SPEAKER:
Harel Shein - Senior Engineering Manager, Datadog

πŸ‘‰ Sign up for our "No BS" Newsletter to get the latest technical data & AI content: aicouncil.com/newsletter

ABOUT AI COUNCIL:
AI Council brings together the brightest minds in data to share industry knowledge, technical architectures and best practices in building cutting edge data & AI systems and tools.

FIND US:
Website: aicouncil.com
LinkedIn: linkedin.com/company/aicouncilconf
X: https://x.com/aicouncilconf
Five years of OpenLineage: How we built an industry standard and why agents need it | DatadogBuilding Durable, Long-Running Autonomous Agents | RedScope AIAI Needs a New Kind of OLTP: Lakebase & Serverless Postgres in the Agent Era | DatabricksWhat are exciting new computer vision use cases with VLMs? | Vik Korrapati, MoondreamWhen Do AI Coding Agents Need a Human in the Loop?Your Database Wasnt Built for This | CockroachDBLook Ma, No Data Warehouse!Why GPUs Keep Getting Faster While CPUs Stalled - And What It Means for Your CodeWrite Less More: How Dagster Rebuilt Our Docs From the Ground UpWe Solved Agent Building | VercelEngineering Earths Largest Biological Data PipelineTrillions-to-Reality: How SambaNova’s Memory-Centric Design Powers Agentic AI & GenAI Workflows
AI Council |

Five years of OpenLineage: How we built an industry standard and why agents need it | Datadog

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER