From Testing to Reliability: Detecting Silent Failures in Production Systems @TeamNANOG
From Testing to Reliability: Detecting Silent Failures in Production Systems  @TeamNANOG
Uploaded October 2026 | Updated November 2026, 15 hours ago
Traditional monitoring focuses on infrastructure health metrics such as uptime, latency, error rates, and resource utilization. While these indicators are essential for operational visibility, they do not always reveal when systems begin producing incorrect outcomes after configuration or logic changes. In complex production environments, services may appear fully operational while automated outputs silently drift from expected behavior.

This presentation explores how reliability practices can evolve from traditional pre-release testing toward continuous validation of production behavior. As systems increasingly rely on configuration-driven logic, rule engines, and distributed service workflows, detecting these silent failures requires observability beyond infrastructure metrics.

The talk examines common failure patterns where configuration changes, rule updates, or dependency interactions alter system behavior without triggering infrastructure alerts. It introduces practical techniques for improving change confidence in production systems, including monitoring automated outcomes, validating outputs against expected baselines, using synthetic production traffic for continuous verification, and detecting statistical anomalies in service behavior.

Attendees will gain practical insights into how reliability teams can identify hidden failure modes earlier, extend observability beyond traditional infrastructure metrics, and strengthen change validation practices in complex production systems.
From Testing to Reliability: Detecting Silent Failures in Production SystemsBuilding Community Owned Critical InfrastructureWhen LLMs Meet ASNs: Rethinking AS-to-Organization Mapping at Internet ScaleExodus of IPv4 from War-torn UkraineNANOG 98 Conference ClosingLightning Talk: The Aisuru Botnet - The New Age of SuckNANOG 96 Conference OpeningSmarter, Faster, Harder: Defending Networks Against AI-Enhanced AttacksRNG: Flat Datacenter Networks at ScaleFinancial Analysis for Technical ProfessionalsARIN UpdateEVPN Interconnect Architecture for Resilient Multi-Domain Fabrics
NANOG |

From Testing to Reliability: Detecting Silent Failures in Production Systems

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER