Uploaded September 2025 | Updated September 2026, 4 hours ago
(Jenya Bogacheva) Sometimes you inherit a clunky pipeline. Sometimes an LLM writes one for you. Either way, you’re stuck with something slow, memory-hungry, and hard to scale.
This talk is about what happens next — how to turn a naive tabular data pipeline into something fast, efficient, and scalable. You’ll get a guided tour through a zoo of optimization techniques: reducing algorithmic complexity, minimizing memory usage, improving I/O throughput, and swapping in Polars — a fast, Rust-based DataFrame library — in place of Pandas (for reasons beyond just hype). By walking through a real-world example step by step, you’ll see how each change makes an impact — and come away with a sharper eye for spotting similar bottlenecks or inefficiencies in your own pipelines.
The walkthrough is grounded in a real-world ML feature engineering task from the aviation industry. But in the spirit of spring, we’ll swap baggage belts for bird feeders — and reframe the problem through a birdwatcher’s lens, not by tracking airport operations, but by counting sparrows and mynas visiting my backyard feeder.
pretalx.com/pycon-au-2025/talk/FHMZBD
python, pycon, australia, programming, conference, technical, developers, panel, sessions, libraries, frameworks, community, sysadmins, students, education, data, science
Videos licensed as CC-BY-NC-SA 4.0
PyCon AU is the national conference for the Python programming community, bringing together professional, student and enthusiast developers, sysadmins and operations folk, students, educators, scientists, statisticians, and many others besides, all with a love for working with Python.
Licensed as CC BY-NC-SA - creativecommons.org/licenses/by-nc-sa/4.0
Produced by Next Day Video Australia: https://nextdayvideo.com.au
Sun Sep 14 11:20:00 2025 at Ballroom 2
(Jenya Bogacheva) Sometimes you inherit a clunky pipeline. Sometimes an LLM writes one for you. Either way, you’re stuck with something slow, memory-hungry, and hard to scale.
This talk is about what happens next — how to turn a naive tabular data pipeline into something fast, efficient, and scalable. You’ll get a guided tour through a zoo of optimization techniques: reducing algorithmic complexity, minimizing memory usage, improving I/O throughput, and swapping in Polars — a fast, Rust-based DataFrame library — in place of Pandas (for reasons beyond just hype). By walking through a real-world example step by step, you’ll see how each change makes an impact — and come away with a sharper eye for spotting similar bottlenecks or inefficiencies in your own pipelines.
The walkthrough is grounded in a real-world ML feature engineering task from the aviation industry. But in the spirit of spring, we’ll swap baggage belts for bird feeders — and reframe the problem through a birdwatcher’s lens, not by tracking airport operations, but by counting sparrows and mynas visiting my backyard feeder.
pretalx.com/pycon-au-2025/talk/FHMZBD
python, pycon, australia, programming, conference, technical, developers, panel, sessions, libraries, frameworks, community, sysadmins, students, education, data, science
Videos licensed as CC-BY-NC-SA 4.0
PyCon AU is the national conference for the Python programming community, bringing together professional, student and enthusiast developers, sysadmins and operations folk, students, educators, scientists, statisticians, and many others besides, all with a love for working with Python.
Licensed as CC BY-NC-SA - creativecommons.org/licenses/by-nc-sa/4.0
Produced by Next Day Video Australia: https://nextdayvideo.com.au
Sun Sep 14 11:20:00 2025 at Ballroom 2
