A Practical Guide to LLM Evaluation - Michelle Yi @ODSCAI
A Practical Guide to LLM Evaluation - Michelle Yi  @ODSCAI
Uploaded April 2026 | Updated November 2026, 4 hours ago
As organizations race to integrate Large Language Models (LLMs) into products and workflows, the challenge of robust evaluation has become a critical bottleneck. Standard academic benchmarks often fail to predict real-world performance, leaving teams unsure if their models are reliable, safe, or truly effective for their specific use case and leaving open the question of business value. This gap between benchmark scores and operational utility introduces significant risks, from brand-damaging hallucinations and biased outputs to poor user experiences that erode trust.

This session cuts through the noise to provide a hands-on, practical guide for product, business, and technical practitioners tasked with validating LLM-powered systems. We will explore a comprehensive evaluation toolkit that balances cost, speed, and quality. We'll cover the pros and cons of automated metrics (e.g., ROUGE, BLEU), the strategic use of LLM-as-judge frameworks for nuanced qualitative assessment, and best practices for designing efficient and scalable human-in-the-loop evaluation pipelines.

-------------------------------------------------------------------------------------------------------------

Visit our website and choose the nearest ODSC AI event to attend and experience all our training and workshops: odsc.ai

To watch more videos like this, visit https://aiplus.training

Sign up for the newsletter to stay up to date with the latest trends in data science: opendatascience.com/newsletter

Follow us online!
• Facebook: facebook.com/OPENDATASCI
• Instagram: instagram.com/odsc
• Blog: opendatascience.com
• LinkedIn: linkedin.com/company/open-data-science
• X (twitter): https://x.com/_odsc
A Practical Guide to LLM Evaluation - Michelle YiLast Week in AI: Long-Running Agents, New AI Models & Evolving AI Skills | Oct 5–11, 2026Build the skills behind practical Personal AI Agents!Devin AI: The $10B AI Engineer Taking Over TechThe Data Scientist’s Edge in the Age of AI with Sebastian WernickeAI Agents Are REVOLUTIONIZING Engineering! You Wont Believe ItSecuring AI Agents: Risk, Governance, Recovery, and Anthropic’s Mythos with Arvind NithrakashyapFrom AI User to AI Builder: The Skills You Need to Build AI Agents and WorkflowsJoin us in New York on August 27 for an ODSC AI Skills AcceleratorAI Attacks Are Moving at Machine Speed. Defense Has to Catch Up with David CampbellAI Agents: The Models Are Ready. The Systems Arent with Scott AskinosieBuilding the AI Ecosystem in Africa: Innovation, Community and the Future of Intelligence
Open Data Science and AI Conference |

A Practical Guide to LLM Evaluation - Michelle Yi

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER