Scaling CDC to Trillions of Rows: What Broke, What We Rebuilt, and What AI Demands Next | Artie @aicouncilconf
Scaling CDC to Trillions of Rows: What Broke, What We Rebuilt, and What AI Demands Next | Artie  @aicouncilconf
Uploaded June 2026 | Updated November 2026, 1 day ago
[2026 - DAY 1 - DATA ENG & DATABASES] Most CDC pipelines work fine when you're building an MVP. Ours did too - until they didn't. Artie is a real-time data replication platform that processes 20-30 billion events per day across thousands of pipelines with sub-minute latency on 90% of them. Three years ago we were running a forked version of Debezium with Kafka processing millions of rows. Along the way, many assumptions we started with broke.

This talk is a post-mortem of what failed, what we rebuilt, and the decisions that matter at scale:

• Why we replaced Debezium - single-threaded capture, limited extensibility, and no built-in recovery forced us to build a proprietary Reader from scratch to increase fault tolerance
• Parallel backfills without data loss - running historical loads alongside live CDC using primary-key range chunking and exactly-once merge semantics, following Netflix's DBLog pattern
• Fan-in from thousands of single-tenant databases - consolidating sharded or single-tenant sources into unified destination schemas without bespoke ETL per tenant
• Edge cases at scale - five-digit-year timestamps, negative years, non-JSON in JSONB, non-UTF8 encodings, and why we chose to fail hard rather than silently skip data (and the recovery mechanisms that make that practical in production)
• Schema evolution - automatic column adds, type changes, drops, and notifications so teams know what changed

Finally: AI workloads have the same freshness problem databases have always had, but the sources are no longer just databases - they are filesystems, object stores, git repos, and documents. We will share how Artie is extending its core primitives beyond databases to become the sync layer for any data AI systems depend on.

Viewers will leave with concrete architectural patterns for building CDC systems that survive at scale, a checklist of failure modes, and a framework for thinking about real-time data as AI infrastructure.

SPEAKER:
Robin Tang - Co-founder & CTO, Artie

👉 Sign up for our "No BS" Newsletter to get the latest technical data & AI content: aicouncil.com/newsletter

ABOUT AI COUNCIL:
AI Council brings together the brightest minds in data to share industry knowledge, technical architectures and best practices in building cutting edge data & AI systems and tools.

FIND US:
Website: aicouncil.com
LinkedIn: linkedin.com/company/aicouncilconf
X: https://x.com/aicouncilconf
Scaling CDC to Trillions of Rows: What Broke, What We Rebuilt, and What AI Demands Next | ArtieHow Action Bias Breaks Autonomous Software Maintenance | LogicStar.aiDo the Boring Stuff to Make Open Source AI Win | Mozilla.aiQ&A with Chang She, CEO, LanceDB: Why the 20-year-old data stack is breaking under AI workloadsForgiveness, Not Permission: Running Agents On Production Data | BauplanInstant Preview Mode  Real Time Feedback to Make SQL Data Exploration FlyEmbedding OLAP, Everywhere  Lessons from OktaGreat Infra for Background Agents | SailAI Launchpad 2026: LogicStar AIPricing AI Agents | OrbWhat Happens to BI in an AI-First World? | EvidenceDuckDB + PostgreSQL: Building a Data Warehouse Inside Your Favorite Database
AI Council |

Scaling CDC to Trillions of Rows: What Broke, What We Rebuilt, and What AI Demands Next | Artie

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER