Uploaded July 2026 | Updated September 2026, 3 weeks ago
Most AI demos look promising. Far fewer make it into production with measurable ROI.
In this Observe 2026 session, CVS Health shares what it takes to move AI initiatives from demos and stalled pilots into durable, production-grade systems. The core message: model capability is not the bottleneck. The real work is building the discipline around the model — evaluation, governance, deployment, observability, economics, and measurable business outcomes.
You’ll learn:
- Why better models do not automatically create durable AI ROI
- How AI initiatives get stuck in “pilot purgatory”
- The three layers of an AI-native SDLC: personal productivity, collaboration/process, and validation/observability
- Why teams need shared tools, reusable practices, clear specs, and documented tribal knowledge
- Why evals are to GenAI what unit tests are to software engineering
- Why cost per outcome matters more than cost per token
- How to tie AI spend to business KPIs like cycle time, throughput, error rate, and time to resolution
- Why explainability, audit trails, and guardrails are essential in regulated environments
- How to build AI systems that can be trusted, measured, governed, and stopped when they are not working
Chapters:
0:00 From demos to durable ROI
0:58 CVS Health’s AI production journey
1:30 Real gains from AI in production
2:12 The bottleneck isn’t the model
3:04 Pilot purgatory is a discipline problem
3:28 Three layers of the AI-native SDLC
4:54 Layer 1: personal productivity
6:05 Where AI coding tools fail
7:23 Standardizing tools so practices compound
7:50 Layer 2: collaboration and process
9:18 When individual speed creates team bottlenecks
10:03 Layer 3: validation and observability
11:26 User validation, autonomy, and compliance risk
12:50 Three disciplines for durable ROI
13:43 Evals are unit tests for GenAI
14:24 Cost per outcome beats cost per token
15:00 Tie AI spend to business KPIs
15:37 Data hygiene and durable infrastructure
16:26 Automate the system, not the step
17:32 Explainability and governance
18:30 Guardrails in regulated environments
20:53 Final takeaways: measuring durable AI ROI
Presented at Observe 2026.
#AIAgents #EnterpriseAI #CVSHealth
🔗 Try Arize AX & Phoenix OSS: arize.com
🔔 Subscribe for weekly content on LLMs, agents, and evaluation: youtube.com/@arizeai?sub_confirmation=1
Most AI demos look promising. Far fewer make it into production with measurable ROI.
In this Observe 2026 session, CVS Health shares what it takes to move AI initiatives from demos and stalled pilots into durable, production-grade systems. The core message: model capability is not the bottleneck. The real work is building the discipline around the model — evaluation, governance, deployment, observability, economics, and measurable business outcomes.
You’ll learn:
- Why better models do not automatically create durable AI ROI
- How AI initiatives get stuck in “pilot purgatory”
- The three layers of an AI-native SDLC: personal productivity, collaboration/process, and validation/observability
- Why teams need shared tools, reusable practices, clear specs, and documented tribal knowledge
- Why evals are to GenAI what unit tests are to software engineering
- Why cost per outcome matters more than cost per token
- How to tie AI spend to business KPIs like cycle time, throughput, error rate, and time to resolution
- Why explainability, audit trails, and guardrails are essential in regulated environments
- How to build AI systems that can be trusted, measured, governed, and stopped when they are not working
Chapters:
0:00 From demos to durable ROI
0:58 CVS Health’s AI production journey
1:30 Real gains from AI in production
2:12 The bottleneck isn’t the model
3:04 Pilot purgatory is a discipline problem
3:28 Three layers of the AI-native SDLC
4:54 Layer 1: personal productivity
6:05 Where AI coding tools fail
7:23 Standardizing tools so practices compound
7:50 Layer 2: collaboration and process
9:18 When individual speed creates team bottlenecks
10:03 Layer 3: validation and observability
11:26 User validation, autonomy, and compliance risk
12:50 Three disciplines for durable ROI
13:43 Evals are unit tests for GenAI
14:24 Cost per outcome beats cost per token
15:00 Tie AI spend to business KPIs
15:37 Data hygiene and durable infrastructure
16:26 Automate the system, not the step
17:32 Explainability and governance
18:30 Guardrails in regulated environments
20:53 Final takeaways: measuring durable AI ROI
Presented at Observe 2026.
#AIAgents #EnterpriseAI #CVSHealth
🔗 Try Arize AX & Phoenix OSS: arize.com
🔔 Subscribe for weekly content on LLMs, agents, and evaluation: youtube.com/@arizeai?sub_confirmation=1
