Build Trustworthy LLM Apps Powered by Agentic Evals | Meta @aicouncilconf
Build Trustworthy LLM Apps Powered by Agentic Evals | Meta  @aicouncilconf
Uploaded June 2026 | Updated November 2026, 1 day ago
[2026 - DAY 3 - LIGHTNING TALK] Agentic AI has certainly secured mindspace of industry. As more & more people continue to play with LLMs, they begin developing an intuition aka vibes around whether LLM is performing the task at hand well or not. While usecases like math problem, classification are easier to verify; generative content creation etc are not. Adding agents to the mix makes things lot more complex. This talk will highlight the particular challenges one may face as they continue their journey towards productionizing their Agentic application: outputs are non deterministic, ground truth is hard to find, and what used to work before no longer does. Yet """"LGTM"""" isn't a deployment strategy.

While typical LLM chatbot can be seen as a more sophisticated Google search, we're beginning to expect more from Agents: Don't just give an answer but actually perform the task. Security, bias, privacy etc suddenly become non-negotiable. Only after we handle these complexities, we can unblock real usecases in healthcare, finance, legal etc.

This talk tackles why agent evaluation is fundamentally harder than traditional ML testing: multi-step reasoning chains, tool use side effects & more. How to build evaluation datasets that actually reflect production scenarios, not just cherry-picked examples. We'll cover automated evaluation pipelines using LLM-as-judge patterns, and when you can not avoid human in the loop. The session addresses detecting regressions before users do: setting up continuous evaluation that catches model degradation. Tricky cases when agent aces public evals but fails in production, and how to build evaluations that predict real-world performance.

SPEAKER:
Rushabh Mehta - Tech Lead, Meta

πŸ‘‰ Sign up for our "No BS" Newsletter to get the latest technical data & AI content: aicouncil.com/newsletter

ABOUT AI COUNCIL:
AI Council brings together the brightest minds in data to share industry knowledge, technical architectures and best practices in building cutting edge data & AI systems and tools.

FIND US:
Website: aicouncil.com
LinkedIn: linkedin.com/company/aicouncilconf
X: https://x.com/aicouncilconf
Build Trustworthy LLM Apps Powered by Agentic Evals | MetaThe Agent Attack Surface: Why AI Is Breaking Software Security As We Know It | SocketData Lake CDC: Are we there yet? | ClickHouseFrom 100 QPS to 100,000: How Agents Are Breaking the Database | Chang She (LanceDB)The End of the Internet As We Know ItThe Mythical Agent-Month | PositBuilding vision AI that works in the wild | Vik Korrapati, MoondreamNo Dropped Frames: designing a VLM around a latency budget | MoondreamBeyond Pipelines: Introducing Pub Sub for TablesTrinity: Training a 400B MoE from Scratch Without Losing Your Mind | ArceeA Guide to Deploy an Enterprise-Ready ClawdBot in Under 30 Minutes | TextQLHow to Unlock Enterprise Value by Training Your Own Language Models | Snowflake
AI Council |

Build Trustworthy LLM Apps Powered by Agentic Evals | Meta

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER