How Microsoft’s Azure AI Foundry Builds Trustworthy AI Agents, with Phoenix @arizeai

Uploaded July 2025 | Updated September 2026, 2 weeks ago

Join Aditi Maheshwari (Senior Product Manager, Microsoft) and Farzad Sunavala (Principal Product Manager, Microsoft) for a walkthrough of Azure AI Foundry—Microsoft’s unified platform to design, customize, and manage multi-agent AI applications.

In this fast-paced session, Aditi walks through live code and shows how to:
Choose and customize models like GPT-4o
Combine Azure AI Search with Foundry Agent Services
Run hybrid and semantic search with RRF for accurate results
Trace agents using Phoenix, an open-source observability library

Chapters:
0:00 Introduction by Aditi Maheshwari
0:29 Why Agentic AI is the next wave
1:01 Azure AI Foundry overview
1:47 Choosing the right model (OpenAI, Meta, etc.)
2:20 Customization with Azure AI Search and Blob Storage
3:45 Vector search vs hybrid search vs semantic re-ranking
5:12 Integrating Phoenix for agent observability
6:08 Tool definitions and multi-agent setup
7:10 Defining agents and prompt-based behavior
8:00 Live agent demo with Phoenix traces
8:40 Comparing Arize and Phoenix for observability
9:13 Closing and additional resources

🎯 Use cases covered: Enterprise RAG, knowledge management, and evaluation of agent workflows
📦 Built-in support for custom tools, interoperability, and secure deployment

📘 Learn more about Arize Phoenix: docs.arize.com/phoenix
🎟️ Attend upcoming Arize AI events: arize.com/community

How Microsoft’s Azure AI Foundry Builds Trustworthy AI Agents, with Phoenix

AI Agent Mastery Certification Course: Lab 4 – Tools & MCP

LLM-as-a-Judge for Agents: How to Build a Custom Eval Rubric That Works | Ep. 7

Why AI Agents Need Their Own Observability Layer | AWS | Arize Observe 2026

Upstart’s First AI Voice Bot: Lessons From Production | Shiv Indap | Arize Observe 2026

Analyzing LLM Evaluations of Customer Reviews Using Repetitions Feature

How to Build Self-Improving AI Agents with Coding Agents | Ep. 13

$How Uber Evaluates AI Agents at Production Scale | Arize Observe 2026 Running evals is one thing. Building an evaluation system that uncovers problems, changes the roadmap, and continuously improves an AI agent is much harder. In this Arize:Observe 2026 session, Aayush Agrawal, Senior AI Product Manager at Uber, explains what Uber learned while building an evaluation platform for production AI agents. Uber’s agent platform supports teams ranging from first-time agent builders to engineers shipping customer-facing agents at global scale. Across those teams, Uber repeatedly found that access to evaluation tools was not enough. Teams needed tracing by default, automatically generated evaluators, continuously updated datasets, broader ownership, and a development process built around learning from production. Aayush covers: • Why teams tend to build their agents first and add evals later • How Uber provides tracing automatically when an agent is deployed • Why complete agent trajectories matter more than input-output logs • How Uber generates agent-specific evaluators from configurations and traces • Turning production failures into continuously updated evaluation datasets • Using real conversations to simulate multi-turn agent behavior • Giving product managers, designers, and operations teams ownership of evals • Why optimizing for a single launch score can produce misleading results • The questions Uber uses to assess whether an evaluation system is useful • How a voice-booking agent exposed a failure that offline evals missed • Using changes in session length to detect unexpected production behavior • How production traces can power agent insights, experiments, and improved versions One example came from Uber’s voice-booking agent. When a child mentioned wanting pizza during a ride request, the agent interpreted the background speech as a new destination. Offline evals had not anticipated the scenario, but a spike in the average number of conversation turns surfaced the problem. A conversational designer then used that insight to update the agent, evaluators, and dataset. The larger lesson is that evals should help teams learn what to build next. The most effective systems connect production traces, failure analysis, datasets, experiments, and agent improvements in one continuous loop. :contentReference[oaicite:0]{index=0} Chapters: 00:00 Why having evals is not enough 00:48 The stakes of running AI agents at Uber 01:40 Inside Uber’s agent platform 02:41 Supporting every type of agent builder 03:38 Why teams add evals too late 04:25 Making tracing the default 05:48 Automatically generating useful evaluators 07:03 Building datasets from production failures 08:20 Bringing product and design teams into evals 09:37 Why launch-gate metrics fail 10:09 Better questions for evaluating your evals 12:01 What Uber’s voice-booking agent taught the team 13:12 From eval afterthought to product insight engine 13:36 The future agent improvement loop 15:01 The hardest part was never the tooling 🔗 Learn more about Arize: https://arize.com 🔗 Explore Arize:Observe: https://arize.com/observe 🔔 Subscribe for more talks about AI agents, LLM evaluation, observability, and production AI: https://www.youtube.com/@arizeai?sub_confirmation=1 #AIAgentEvals #Uber #AIEngineering$

Identity, Permissions, and Security for AI Agents | WorkOS | Arize Observe 2026

How My AI Agent Rewrites Itself Overnight | Chi Wang, AG2 | Arize Observe 2026