Uploaded August 2026 | Updated September 2026, 21 minutes ago
(Wesley Volz) Many teams building AI products focus on parsing new context and editing prompts, without knowing if things are improving. They then learn that even a perfect model won't drive success if the product isn't easy to use, or people abandon it after one attempt. Getting the model right is necessary but on its own isn't sufficient.
This talk is about the measurement gap between a model that performs well in offline evaluation and an AI product that actually benefits users. Using a real-world case study, I'll walk through how we built three layers of measurement that gave us genuine confidence in product decisions and shaped team priorities.
These three layers - offline model evaluation, online usage metrics, and qualitative user feedback - don't always agree. I'll share why human judgement is essential to make key decisions when navigating conflicting signals, and how to do this with confidence.
If you're building AI products and can't answer "is this actually getting better?" - this talk is for you.
https://2026.pycon.org.au/schedule/9AK3WS/
python, pycon, australia, programming, conference, technical, developers, panel, sessions, libraries, frameworks, community, sysadmins, students, education, data, science
Videos licensed as CC-BY-NC-SA 4.0
PyCon AU is the national conference for the Python programming community, bringing together professional, student and enthusiast developers, sysadmins and operations folk, students, educators, scientists, statisticians, and many others besides, all with a love for working with Python.
Licensed as CC BY-NC-SA - creativecommons.org/licenses/by-nc-sa/4.0
Produced by Next Day Video Australia: https://nextdayvideo.com.au
Sat Aug 29 14:35:00 2026 AEST (UTC+10) at Ballroom 2
(Wesley Volz) Many teams building AI products focus on parsing new context and editing prompts, without knowing if things are improving. They then learn that even a perfect model won't drive success if the product isn't easy to use, or people abandon it after one attempt. Getting the model right is necessary but on its own isn't sufficient.
This talk is about the measurement gap between a model that performs well in offline evaluation and an AI product that actually benefits users. Using a real-world case study, I'll walk through how we built three layers of measurement that gave us genuine confidence in product decisions and shaped team priorities.
These three layers - offline model evaluation, online usage metrics, and qualitative user feedback - don't always agree. I'll share why human judgement is essential to make key decisions when navigating conflicting signals, and how to do this with confidence.
If you're building AI products and can't answer "is this actually getting better?" - this talk is for you.
https://2026.pycon.org.au/schedule/9AK3WS/
python, pycon, australia, programming, conference, technical, developers, panel, sessions, libraries, frameworks, community, sysadmins, students, education, data, science
Videos licensed as CC-BY-NC-SA 4.0
PyCon AU is the national conference for the Python programming community, bringing together professional, student and enthusiast developers, sysadmins and operations folk, students, educators, scientists, statisticians, and many others besides, all with a love for working with Python.
Licensed as CC BY-NC-SA - creativecommons.org/licenses/by-nc-sa/4.0
Produced by Next Day Video Australia: https://nextdayvideo.com.au
Sat Aug 29 14:35:00 2026 AEST (UTC+10) at Ballroom 2
