Uploaded September 2025 | Updated September 2026, 9 hours ago
(Mimoune DJOUALLAH) In this talk, we’ll explore four modern data engines ; Daft, DuckDB, Polars, and DataFusion, that offer varying levels of out-of-core execution. These engines allow you to work with datasets larger than memory without the need to rewrite everything for a distributed system.
They’re fast, expressive, and, above all, Pythonic—though SQL support might still be the deciding factor for many workflows.
Rather than comparing which engine is best (they’re all open source, and good ideas tend to spread quickly), this talk will highlight exciting recent developments. We’ll showcase, for example, how one workload saw a 2× performance boost in under a year, and how open table formats are bringing cloud data warehouse capabilities to local Python workflows.
pretalx.com/pycon-au-2025/talk/ECVWJU
python, pycon, australia, programming, conference, technical, developers, panel, sessions, libraries, frameworks, community, sysadmins, students, education, data, science
Videos licensed as CC-BY-NC-SA 4.0
PyCon AU is the national conference for the Python programming community, bringing together professional, student and enthusiast developers, sysadmins and operations folk, students, educators, scientists, statisticians, and many others besides, all with a love for working with Python.
Licensed as CC BY-NC-SA - creativecommons.org/licenses/by-nc-sa/4.0
Produced by Next Day Video Australia: https://nextdayvideo.com.au
Fri Sep 12 09:20:00 2025 at Ballroom 1
(Mimoune DJOUALLAH) In this talk, we’ll explore four modern data engines ; Daft, DuckDB, Polars, and DataFusion, that offer varying levels of out-of-core execution. These engines allow you to work with datasets larger than memory without the need to rewrite everything for a distributed system.
They’re fast, expressive, and, above all, Pythonic—though SQL support might still be the deciding factor for many workflows.
Rather than comparing which engine is best (they’re all open source, and good ideas tend to spread quickly), this talk will highlight exciting recent developments. We’ll showcase, for example, how one workload saw a 2× performance boost in under a year, and how open table formats are bringing cloud data warehouse capabilities to local Python workflows.
pretalx.com/pycon-au-2025/talk/ECVWJU
python, pycon, australia, programming, conference, technical, developers, panel, sessions, libraries, frameworks, community, sysadmins, students, education, data, science
Videos licensed as CC-BY-NC-SA 4.0
PyCon AU is the national conference for the Python programming community, bringing together professional, student and enthusiast developers, sysadmins and operations folk, students, educators, scientists, statisticians, and many others besides, all with a love for working with Python.
Licensed as CC BY-NC-SA - creativecommons.org/licenses/by-nc-sa/4.0
Produced by Next Day Video Australia: https://nextdayvideo.com.au
Fri Sep 12 09:20:00 2025 at Ballroom 1
