Uploaded September 2025 | Updated September 2026, 2 weeks ago
Notebooks for homeworks:
github.com/Arize-ai/phoenix/tree/main/tutorials/ai_evals_course
This alternate third extra credit homework featuring Arize Phoenix is part of a series developed by the Arize AI team for the AI Evals For Engineers & PMs course by Parlance Labs.
Build an LLM-as-a-Judge in Homework 3 of our AI Evaluations module using Phoenix (open source) to assess a recipe bot’s outputs against dietary restrictions (vegan, Whole30, diabetic-friendly, gluten-free, paleo). We spin up Phoenix, trace OpenAI calls, create & compare ground truths, run experiments (accuracy, TP/TN, confusion matrix), analyze failure cases (e.g., honey/whole-wheat/peas), and iteratively refine the evaluator with definitions + few-shot examples—ending with strong test accuracy and end-to-end trace evaluations.
Notebooks for homeworks:
github.com/Arize-ai/phoenix/tree/main/tutorials/ai_evals_course
This alternate third extra credit homework featuring Arize Phoenix is part of a series developed by the Arize AI team for the AI Evals For Engineers & PMs course by Parlance Labs.
Build an LLM-as-a-Judge in Homework 3 of our AI Evaluations module using Phoenix (open source) to assess a recipe bot’s outputs against dietary restrictions (vegan, Whole30, diabetic-friendly, gluten-free, paleo). We spin up Phoenix, trace OpenAI calls, create & compare ground truths, run experiments (accuracy, TP/TN, confusion matrix), analyze failure cases (e.g., honey/whole-wheat/peas), and iteratively refine the evaluator with definitions + few-shot examples—ending with strong test accuracy and end-to-end trace evaluations.
