Aligning LLM Evaluators with Human Annotations (using Mastra agents) @arizeai

Uploaded September 2025 | Updated September 2026, 2 weeks ago

Evals are everywhere, but generic benchmarks rarely capture the nuances of your application. In this tutorial, you’ll learn how to build custom LLM evaluators that align with your specific use case by incorporating human feedback and running a clear, repeatable workflow.

Everything is implemented in TypeScript, with code examples.

Code: arize.com/docs/phoenix/cookbook/human-in-the-loop-workflows-annotations/aligning-llm-evals-with-human-annotations-typescript
Phoenix: arize.com/docs/phoenix
Community: arize.com/community

Aligning LLM Evaluators with Human Annotations (using Mastra agents)

From AI Coding Agents to the Software Factory | Factory AI | Arize Observe 2026

One AI Question - why do you use an LLM to evaluate another LLM with Ankur Duggal

$When AI Can Write Code, What Are Software Engineers Worth? | Citadel When AI can generate code in seconds, what still makes a software engineer valuable? In this Arize:Observe session, Craig Owenby of Citadel explores how agentic coding tools are changing software engineering, and why the profession still requires far more than producing code. Craig compares large language models to the printing press. The printing press replaced the manual work of copying books, but it did not replace authors. In the same way, AI coding agents can automate the mechanics of writing code without replacing the judgment, vision, empathy, and experience required to build useful software. The session covers: • Why “coder” and “software engineer” are increasingly different roles • How tools like Claude Code, Codex, Copilot, and Cursor remove traditional barriers to building software • What the printing press teaches us about AI-assisted development • Why engineers should avoid competing with AI on raw code generation • How Sears lost its advantage by competing with e-commerce on the wrong terms • Why human experience, intuition, and empathy remain essential • How constraints can improve product and engineering decisions • Why shipping more features can increase volatility and reduce user trust • How AI acts as leverage for strong and weak engineering decisions • Why product direction and problem selection matter more as implementation gets easier The central lesson: software engineering is not primarily about writing code. It is about deciding what should be built, understanding why it matters, and applying technology with judgment. When everyone can code, engineers differentiate themselves through their standards, instincts, product sense, and ability to understand the people using what they build. :contentReference[oaicite:0]{index=0} Chapters: 00:00 When everyone can code, what are engineers worth? 00:46 LLMs and the printing press 01:30 The limits that shaped software engineering 02:42 We finally live in a world where everyone can code 03:45 The existential question for experienced engineers 04:45 Coders versus software engineers 06:05 Don’t make the same mistake as Sears 07:38 Human judgment, experience, and empathy 09:02 Why constraints can produce better software 10:25 Product volatility and the Sharpe ratio 11:49 Software engineering is about solving problems 12:38 LLMs as leverage for engineers 13:45 What sets engineers apart 14:25 Your humanity is the key 🔗 Learn more about Arize: https://arize.com 🔗 Explore Arize:Observe: https://arize.com/observe 🔔 Subscribe for more talks on AI engineering, coding agents, evaluation, observability, and the future of software development: https://www.youtube.com/@arizeai?sub_confirmation=1 #SoftwareEngineering #CodingAgents #AIEngineering$

One AI Question - how is marketing at Arize using AI, with Chris Cooning

Salesforce - Monitoring, Analyzing, and Improving Multi Agent Implementation

Source Allies - Generative AI Agents Jack of All Trades or Master of One

Why Your LLM Starts Forgetting Instructions

The Emerging Stack for AI Compute: Kubernetes + Ray + PyTorch + vLLM

Introducing Strands Agents: AWS on How To Improve the Customer Experience with AI Agents

AI Agent Mastery Certification Course: Module 4 – Tools & MCP

Arize AI |

Aligning LLM Evaluators with Human Annotations (using Mastra agents)