![The Day the Agent Started Lying (Politely) [PyCon DE & PyData 2026] @PyDataTV](https://i.ytimg.com/vi/vZdMN82NMTQ/maxresdefault.jpg)
Uploaded August 2026 | Updated September 2026, 2 weeks ago
đ Recorded at PyCon DE & PyData 2026, 15.04.2026
2026.pycon.de/talks/P8Y9TD
đ Watch Asya Melnik reveal how to detect silent failures and manage data drift to ensure your LLM-based agents remain trustworthy and accurate in production.
Speakers:
Asya Melnik
Description:
LLM agents often suffer from silent degradation, where the model continues to provide confident responses despite a shift in the underlying data distribution. This problem is particularly acute in non-deterministic systems where ground truth labels are unavailable for real-time validation, rendering traditional accuracy metrics useless. In a customer support ticket routing scenario, for example, the launch of a new product can introduce new vocabulary and shift the meaning of existing terms, leading the agent to misclassify ticket priority while reporting zero system errors.
To detect this drift without relying on manual labeling, a multi-signal evaluation framework is used. This approach monitors six distinct metrics: Shannon entropy (calculated using all available class probabilities to measure internal uncertainty), fallback rates (acting as a canary signal), vocabulary drift (measuring the distance of current word vectors from a stable centroid), human disagreement (tracking when users override agent decisions), LLM-as-a-judge (using a model like Claude Haiku to verify if the agent's reasoning is faithful to the input), and trajectory (assessing the logical flow from input to outcome).
The key takeaway is the implementation of a tiered action plan based on the number of triggering signals. A single alert, such as an increased fallback rate, suggests observation. Two or more signals indicate a need for investigation. When multiple signals across different layersâinternal confidence, external human feedback, and logical verificationâtrigger simultaneously, it provides a high-confidence indicator that the agent's prompt or model requires updating. This "nervous system" approach allows operators to identify and fix silent failures before they impact business operations.
âď¸ About PyCon DE:
PyCon DE is the leading conference on open-source Python applications in AI and data science. It brings together industry professionals, researchers, AI and data science practitioners, and software engineering communities, providing a unique platform for collaboration, knowledge sharing, and innovation.
The PyCon DE & PyData 2026 conference delivered an exceptional experience, fostering stronger connections within the Python community while showcasing the latest advancements in artificial intelligence and data science. Attendees enjoyed a diverse and engaging program of talks, workshops, and networking opportunities, further establishing the conference as a premier event for Python, AI, and data science enthusiasts across Germany.
PyCon DE 2027 will take place in Heidelberg from 19 to 23 April 2027.
Follow us:
⢠Newsletter: 2027.pycon.de/newsletter
⢠LinkedIn: linkedin.com/company/pyconde
⢠X: x.com/pyconde
Links:
⢠Conference website: pycon.de
⢠Other sessions: 2026.pycon.de/talks
The conference was organized by
⢠Python Softwareverband e.V.: pysv.org
⢠Pioneers Hub gemeinnßtzige GmbH: pioneershub.org
in collaboration with NumFOCUS Inc.: numfocus.org
If you enjoyed this session, please like, and subscribe to our channel for more insightful talks and discussions.
Share this video with your network to spread the knowledge!
Hashtags:
#Python #PyConDE #PyData #OpenSource #AI #DataScience #MachineLearning #SoftwareEngineering #LLMs #Community #Sovereignty
Acknowledgements:
Special thanks to all the volunteers and sponsors who made this event possible.
About:
Python Softwareverband e.V.:
PySV is a non-profit that promotes the use and development of Python in Germany through events, education, and advocacy, fostering an open Python community.
Pioneers Hub gemeinnĂźtzige GmbH:
is a non-profit fostering innovation in AI and tech by connecting experts and promoting knowledge exchange through events and collaborative initiatives.
NumFOCUS Inc.
supports open-source scientific computing by providing financial and logistical support to key projects like NumPy and Jupyter, promoting sustainable development and collaboration.
pydata.org
PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.
đ Recorded at PyCon DE & PyData 2026, 15.04.2026
2026.pycon.de/talks/P8Y9TD
đ Watch Asya Melnik reveal how to detect silent failures and manage data drift to ensure your LLM-based agents remain trustworthy and accurate in production.
Speakers:
Asya Melnik
Description:
LLM agents often suffer from silent degradation, where the model continues to provide confident responses despite a shift in the underlying data distribution. This problem is particularly acute in non-deterministic systems where ground truth labels are unavailable for real-time validation, rendering traditional accuracy metrics useless. In a customer support ticket routing scenario, for example, the launch of a new product can introduce new vocabulary and shift the meaning of existing terms, leading the agent to misclassify ticket priority while reporting zero system errors.
To detect this drift without relying on manual labeling, a multi-signal evaluation framework is used. This approach monitors six distinct metrics: Shannon entropy (calculated using all available class probabilities to measure internal uncertainty), fallback rates (acting as a canary signal), vocabulary drift (measuring the distance of current word vectors from a stable centroid), human disagreement (tracking when users override agent decisions), LLM-as-a-judge (using a model like Claude Haiku to verify if the agent's reasoning is faithful to the input), and trajectory (assessing the logical flow from input to outcome).
The key takeaway is the implementation of a tiered action plan based on the number of triggering signals. A single alert, such as an increased fallback rate, suggests observation. Two or more signals indicate a need for investigation. When multiple signals across different layersâinternal confidence, external human feedback, and logical verificationâtrigger simultaneously, it provides a high-confidence indicator that the agent's prompt or model requires updating. This "nervous system" approach allows operators to identify and fix silent failures before they impact business operations.
âď¸ About PyCon DE:
PyCon DE is the leading conference on open-source Python applications in AI and data science. It brings together industry professionals, researchers, AI and data science practitioners, and software engineering communities, providing a unique platform for collaboration, knowledge sharing, and innovation.
The PyCon DE & PyData 2026 conference delivered an exceptional experience, fostering stronger connections within the Python community while showcasing the latest advancements in artificial intelligence and data science. Attendees enjoyed a diverse and engaging program of talks, workshops, and networking opportunities, further establishing the conference as a premier event for Python, AI, and data science enthusiasts across Germany.
PyCon DE 2027 will take place in Heidelberg from 19 to 23 April 2027.
Follow us:
⢠Newsletter: 2027.pycon.de/newsletter
⢠LinkedIn: linkedin.com/company/pyconde
⢠X: x.com/pyconde
Links:
⢠Conference website: pycon.de
⢠Other sessions: 2026.pycon.de/talks
The conference was organized by
⢠Python Softwareverband e.V.: pysv.org
⢠Pioneers Hub gemeinnßtzige GmbH: pioneershub.org
in collaboration with NumFOCUS Inc.: numfocus.org
If you enjoyed this session, please like, and subscribe to our channel for more insightful talks and discussions.
Share this video with your network to spread the knowledge!
Hashtags:
#Python #PyConDE #PyData #OpenSource #AI #DataScience #MachineLearning #SoftwareEngineering #LLMs #Community #Sovereignty
Acknowledgements:
Special thanks to all the volunteers and sponsors who made this event possible.
About:
Python Softwareverband e.V.:
PySV is a non-profit that promotes the use and development of Python in Germany through events, education, and advocacy, fostering an open Python community.
Pioneers Hub gemeinnĂźtzige GmbH:
is a non-profit fostering innovation in AI and tech by connecting experts and promoting knowledge exchange through events and collaborative initiatives.
NumFOCUS Inc.
supports open-source scientific computing by providing financial and logistical support to key projects like NumPy and Jupyter, promoting sustainable development and collaboration.
pydata.org
PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.
![The Day the Agent Started Lying (Politely) [PyCon DE & PyData 2026]](https://i.ytimg.com/vi/vZdMN82NMTQ/mqdefault.jpg "The Day the Agent Started Lying (Politely) [PyCon DE & PyData 2026]
đ Recorded at PyCon DE & PyData 2026, 15.04.2026
https://2026.pycon.de/talks/P8Y9TD/
đ Watch Asya Melnik reveal how to detect silent failures and manage data drift to ensure your LLM-based agents remain trustworthy and accurate in production.
Speakers:
Asya Melnik
Description:
LLM agents often suffer from silent degradation, where the model continues to provide confident responses despite a shift in the underlying data distribution. This problem is particularly acute in non-deterministic systems where ground truth labels are unavailable for real-time validation, rendering traditional accuracy metrics useless. In a customer support ticket routing scenario, for example, the launch of a new product can introduce new vocabulary and shift the meaning of existing terms, leading the agent to misclassify ticket priority while reporting zero system errors.
To detect this drift without relying on manual labeling, a multi-signal evaluation framework is used. This approach monitors six distinct metrics: Shannon entropy (calculated using all available class probabilities to measure internal uncertainty), fallback rates (acting as a canary signal), vocabulary drift (measuring the distance of current word vectors from a stable centroid), human disagreement (tracking when users override agent decisions), LLM-as-a-judge (using a model like Claude Haiku to verify if the agents reasoning is faithful to the input), and trajectory (assessing the logical flow from input to outcome).
The key takeaway is the implementation of a tiered action plan based on the number of triggering signals. A single alert, such as an increased fallback rate, suggests observation. Two or more signals indicate a need for investigation. When multiple signals across different layersâinternal confidence, external human feedback, and logical verificationâtrigger simultaneously, it provides a high-confidence indicator that the agents prompt or model requires updating. This nervous system approach allows operators to identify and fix silent failures before they impact business operations.
âď¸ About PyCon DE:
PyCon DE is the leading conference on open-source Python applications in AI and data science. It brings together industry professionals, researchers, AI and data science practitioners, and software engineering communities, providing a unique platform for collaboration, knowledge sharing, and innovation.
The PyCon DE & PyData 2026 conference delivered an exceptional experience, fostering stronger connections within the Python community while showcasing the latest advancements in artificial intelligence and data science. Attendees enjoyed a diverse and engaging program of talks, workshops, and networking opportunities, further establishing the conference as a premier event for Python, AI, and data science enthusiasts across Germany.
PyCon DE 2027 will take place in Heidelberg from 19 to 23 April 2027.
Follow us:
⢠Newsletter: https://2027.pycon.de/newsletter/
⢠LinkedIn: https://www.linkedin.com/company/pyconde
⢠X: https://www.x.com/pyconde
Links:
⢠Conference website: http://pycon.de
⢠Other sessions: https://2026.pycon.de/talks/
The conference was organized by
⢠Python Softwareverband e.V.: http://pysv.org
⢠Pioneers Hub gemeinnßtzige GmbH: http://pioneershub.org
in collaboration with NumFOCUS Inc.: http://numfocus.org
If you enjoyed this session, please like, and subscribe to our channel for more insightful talks and discussions.
Share this video with your network to spread the knowledge!
Hashtags:
#Python #PyConDE #PyData #OpenSource #AI #DataScience #MachineLearning #SoftwareEngineering #LLMs #Community #Sovereignty
Acknowledgements:
Special thanks to all the volunteers and sponsors who made this event possible.
About:
Python Softwareverband e.V.:
PySV is a non-profit that promotes the use and development of Python in Germany through events, education, and advocacy, fostering an open Python community.
Pioneers Hub gemeinnĂźtzige GmbH:
is a non-profit fostering innovation in AI and tech by connecting experts and promoting knowledge exchange through events and collaborative initiatives.
NumFOCUS Inc.
supports open-source scientific computing by providing financial and logistical support to key projects like NumPy and Jupyter, promoting sustainable development and collaboration.
www.pydata.org
PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.")
![Wetterdienst: Fast, Unified Access to Open Weather Data with Polars [PyCon DE & PyData 2026]](https://i.ytimg.com/vi/wegSoXmVYNM/mqdefault.jpg "Wetterdienst: Fast, Unified Access to Open Weather Data with Polars [PyCon DE & PyData 2026]
đ Recorded at PyCon DE & PyData 2026, 15.04.2026
https://2026.pycon.de/talks/V8DNCL/
đ Watch Benjamin Gutzmann demonstrate how to eliminate API complexity and accelerate weather data pipelines using Wetterdienst, a powerful Polars-first library for unified open weather access.
Speakers:
Benjamin
Description:
In this presentation, Benjamin Gutzmann, a Data Engineer at Otto Group data.works, introduces Wetterdienst, a Python library designed to simplify the complex process of accessing open weather data. Because weather and environmental APIs vary wildly in format and structure, data engineers often spend more time on plumbing than on actual analysis. Benjamin explains how Wetterdienst solves this by providing a unified, Polars-first interface that standardizes request patterns across multiple global services, including the DWD, NOAA/NWS, and ECCC.
Viewers will learn how the library normalizes inconsistent data into tidy, long-format DataFrames using SI units and UTC timestamps, while implementing robust caching and retry mechanisms to ensure pipeline reliability. Benjamin walks through the provider architecture and demonstrates practical workflows for station discovery, timeseries retrieval, and exporting data to databases. Whether you are building ETL pipelines or training ML models, this talk provides a blueprint for integrating weather data via Python, CLI, or REST API to accelerate your analytics and operations.
âď¸ About PyCon DE:
PyCon DE is the leading conference on open-source Python applications in AI and data science. It brings together industry professionals, researchers, AI and data science practitioners, and software engineering communities, providing a unique platform for collaboration, knowledge sharing, and innovation.
The PyCon DE & PyData 2026 conference delivered an exceptional experience, fostering stronger connections within the Python community while showcasing the latest advancements in artificial intelligence and data science. Attendees enjoyed a diverse and engaging program of talks, workshops, and networking opportunities, further establishing the conference as a premier event for Python, AI, and data science enthusiasts across Germany.
PyCon DE 2027 will take place in Heidelberg from 19 to 23 April 2027.
Follow us:
⢠Newsletter: https://2027.pycon.de/newsletter/
⢠LinkedIn: https://www.linkedin.com/company/pyconde
⢠X: https://www.x.com/pyconde
Links:
⢠Conference website: http://pycon.de
⢠Other sessions: https://2026.pycon.de/talks/
The conference was organized by
⢠Python Softwareverband e.V.: http://pysv.org
⢠Pioneers Hub gemeinnßtzige GmbH: http://pioneershub.org
in collaboration with NumFOCUS Inc.: http://numfocus.org
If you enjoyed this session, please like, and subscribe to our channel for more insightful talks and discussions.
Share this video with your network to spread the knowledge!
Hashtags:
#Python #PyConDE #PyData #OpenSource #AI #DataScience #MachineLearning #SoftwareEngineering #LLMs #Community #Sovereignty
Acknowledgements:
Special thanks to all the volunteers and sponsors who made this event possible.
About:
Python Softwareverband e.V.:
PySV is a non-profit that promotes the use and development of Python in Germany through events, education, and advocacy, fostering an open Python community.
Pioneers Hub gemeinnĂźtzige GmbH:
is a non-profit fostering innovation in AI and tech by connecting experts and promoting knowledge exchange through events and collaborative initiatives.
NumFOCUS Inc.
supports open-source scientific computing by providing financial and logistical support to key projects like NumPy and Jupyter, promoting sustainable development and collaboration.
www.pydata.org
PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.")
![Solving Marketplace Cold Start at Scale with Ranking [PyCon DE & PyData 2026]](https://i.ytimg.com/vi/xS5xM2ojrH0/mqdefault.jpg "Solving Marketplace Cold Start at Scale with Ranking [PyCon DE & PyData 2026]
đ Recorded at PyCon DE & PyData 2026, 14.04.2026
https://2026.pycon.de/talks/GVHZW9/
đ Watch Theodore Meynard share a pragmatic blueprint for solving the marketplace cold start problem through scalable ranking strategies and experiment-driven iteration.
Speakers:
Theodore Meynard
Description:
Marketplace cold start occurs when new activities lack the historical dataâclicks, bookings, and reviewsârequired by ranking models to achieve visibility, creating a self-reinforcing loop of low exposure. To break this trap and activate the growth flywheel for new supply, a system was developed to transition from rigid, manual constraints to flexible, ML-driven exploration.
Initial attempts focused on prioritizing a queue of new activities using supplier-level features or filtering out low-scoring candidates, both of which yielded marginal results. A breakthrough occurred when the system replaced random selection with the existing ranking model to pick the best candidates for reserved exposure slots. This increased conversion rates and the total number of activated activities. Subsequent iterations removed the queue and prioritization models entirely, allowing all new and unactivated activities to compete in real-time.
The final architecture replaced fixed exposure slots with a blended score, combining the standard ranking score with a boost factor for unactivated items. This flexible boosting approach increased bookings per visitor for unactivated activities. Key findings indicate that removing hand-crafted constraints increases ML optimization and that framing cold start as a supplier activation problem rather than a customer ranking problem provides clearer business metrics for success.
âď¸ About PyCon DE:
PyCon DE is the leading conference on open-source Python applications in AI and data science. It brings together industry professionals, researchers, AI and data science practitioners, and software engineering communities, providing a unique platform for collaboration, knowledge sharing, and innovation.
The PyCon DE & PyData 2026 conference delivered an exceptional experience, fostering stronger connections within the Python community while showcasing the latest advancements in artificial intelligence and data science. Attendees enjoyed a diverse and engaging program of talks, workshops, and networking opportunities, further establishing the conference as a premier event for Python, AI, and data science enthusiasts across Germany.
PyCon DE 2027 will take place in Heidelberg from 19 to 23 April 2027.
Follow us:
⢠Newsletter: https://2027.pycon.de/newsletter/
⢠LinkedIn: https://www.linkedin.com/company/pyconde
⢠X: https://www.x.com/pyconde
Links:
⢠Conference website: http://pycon.de
⢠Other sessions: https://2026.pycon.de/talks/
The conference was organized by
⢠Python Softwareverband e.V.: http://pysv.org
⢠Pioneers Hub gemeinnßtzige GmbH: http://pioneershub.org
in collaboration with NumFOCUS Inc.: http://numfocus.org
If you enjoyed this session, please like, and subscribe to our channel for more insightful talks and discussions.
Share this video with your network to spread the knowledge!
Hashtags:
#Python #PyConDE #PyData #OpenSource #AI #DataScience #MachineLearning #SoftwareEngineering #LLMs #Community #Sovereignty
Acknowledgements:
Special thanks to all the volunteers and sponsors who made this event possible.
About:
Python Softwareverband e.V.:
PySV is a non-profit that promotes the use and development of Python in Germany through events, education, and advocacy, fostering an open Python community.
Pioneers Hub gemeinnĂźtzige GmbH:
is a non-profit fostering innovation in AI and tech by connecting experts and promoting knowledge exchange through events and collaborative initiatives.
NumFOCUS Inc.
supports open-source scientific computing by providing financial and logistical support to key projects like NumPy and Jupyter, promoting sustainable development and collaboration.
www.pydata.org
PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.")
