Uploaded September 2025 | Updated September 2026, 4 hours ago
(Dilpreet Singh) AI can draft product descriptions, handle customer support, and create impressive summaries, yet in production we still struggle to answer a basic question: is this output measurably any good? Traditional accuracy metrics fall short when your LLM needs to write _engaging_ content or your image model is expected to produce _aesthetic_ results. And checking outputs manually? That doesn't scale.
This talk is a friendly introduction to building evaluation loops that work for GenAI models. We'll explore everyday examples such as grading LLM summaries and judging whether chatbot responses help or frustrate, showing why deterministic metrics fail for open-ended outputs. From there, we outline a three-part approach combining simple metrics for quick first-pass evaluation, human-preference samples for nuance, and repeatable tests that run with every model change.
To ground these ideas, we'll walk through a real project that turns elevation data into high-quality Swiss-style relief maps. The domain may seem niche, but the lessons learned of balancing automation with human judgment, tracking non-deterministic outputs, and iterating quickly without drowning in data, apply to every project. You’ll leave with a mental checklist and a starter toolkit for proving that your GenAI output is getting better, not just different.
pretalx.com/pycon-au-2025/talk/E3KDGD
python, pycon, australia, programming, conference, technical, developers, panel, sessions, libraries, frameworks, community, sysadmins, students, education, data, science
Videos licensed as CC-BY-NC-SA 4.0
PyCon AU is the national conference for the Python programming community, bringing together professional, student and enthusiast developers, sysadmins and operations folk, students, educators, scientists, statisticians, and many others besides, all with a love for working with Python.
Licensed as CC BY-NC-SA - creativecommons.org/licenses/by-nc-sa/4.0
Produced by Next Day Video Australia: https://nextdayvideo.com.au
Fri Sep 12 14:30:00 2025 at Ballroom 1
(Dilpreet Singh) AI can draft product descriptions, handle customer support, and create impressive summaries, yet in production we still struggle to answer a basic question: is this output measurably any good? Traditional accuracy metrics fall short when your LLM needs to write _engaging_ content or your image model is expected to produce _aesthetic_ results. And checking outputs manually? That doesn't scale.
This talk is a friendly introduction to building evaluation loops that work for GenAI models. We'll explore everyday examples such as grading LLM summaries and judging whether chatbot responses help or frustrate, showing why deterministic metrics fail for open-ended outputs. From there, we outline a three-part approach combining simple metrics for quick first-pass evaluation, human-preference samples for nuance, and repeatable tests that run with every model change.
To ground these ideas, we'll walk through a real project that turns elevation data into high-quality Swiss-style relief maps. The domain may seem niche, but the lessons learned of balancing automation with human judgment, tracking non-deterministic outputs, and iterating quickly without drowning in data, apply to every project. You’ll leave with a mental checklist and a starter toolkit for proving that your GenAI output is getting better, not just different.
pretalx.com/pycon-au-2025/talk/E3KDGD
python, pycon, australia, programming, conference, technical, developers, panel, sessions, libraries, frameworks, community, sysadmins, students, education, data, science
Videos licensed as CC-BY-NC-SA 4.0
PyCon AU is the national conference for the Python programming community, bringing together professional, student and enthusiast developers, sysadmins and operations folk, students, educators, scientists, statisticians, and many others besides, all with a love for working with Python.
Licensed as CC BY-NC-SA - creativecommons.org/licenses/by-nc-sa/4.0
Produced by Next Day Video Australia: https://nextdayvideo.com.au
Fri Sep 12 14:30:00 2025 at Ballroom 1
