Why data science matters more than ever for AI evaluation @PositPBC
Why data science matters more than ever for AI evaluation  @PositPBC
Uploaded September 2026 | Updated November 2026, 13 hours ago
Speed is no longer the hard part of building with AI. Confidence is. Generative assistants, agents, and copilots let teams go from idea to working prototype in an afternoon, but a working prototype isn't proof that a system behaves the way you expect.

In this talk, James Blair, senior product manager at Posit, makes the case for data scientists as the right people to lead AI evaluation. Unlike traditional software, AI systems can take different paths to different outputs from the same input, so testing for one expected result doesn't work. Data scientists already have the tools for this problem: experimental design, statistical rigor, and the discipline to separate real improvement from random noise.

James walks through what that looks like in practice: evaluating not just a model's final output but the tool calls, sources, and intermediate steps that produced it, building evaluation criteria specific to each system's domain, and treating evaluation as a continuous practice that runs through production, not just development. He also covers how Posit's open-source tooling for R and Python, paired with AWS infrastructure and model access through Bedrock, gives teams a way to build evaluations that scale and hold up over time.

Watch to learn why rigorous evaluation, not raw model capability, decides whether an AI system is ready for production.
See how Posit and AWS work together: posit.co/solutions/aws
Why data science matters more than ever for AI evaluationThe Intersection of Biology, Stats, and CS | Chris Gates | Data Science HangoutExploring Positron settings | Isabel Zimmerman & Davis Vaughan | Data Science LabWhats new in Shiny for Python 1.9?Building Reproducible LLM Workflows | Leslie Emery | Data Science HangoutWhat is the Grammar of Graphics?Enabling SSL SERIES 1/4: Creating an SSL CertHow do you build a Shiny app in 2027? | Dan Chen | Data Science LabHow to use the native pipe in RDriving Shiny adoption by removing friction | Jonathan Lin | Data Science HangoutClinical data science when your patients are kids | Nikolay Braykov | Data Science HangoutMichael Chow: From psychology and Python to constrained creativity
Posit PBC |

Why data science matters more than ever for AI evaluation

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER