Uploaded June 2026 | Updated November 2026, 1 day ago
Abstract
Large-scale network operators running multi-vendor environments face a recurring challenge: how to upgrade network operating systems across thousands of devices without compromising reliability. Upgrade too slowly and you accumulate security exposure, vendor end-of-life risk, and missing capabilities. Upgrade too fast and vendor regressions, configuration incompatibilities, and multi-vendor interaction failures can cause production incidents.
Therefore, managing network OS upgrades across tens and thousands of multi-vendor devices demands a disciplined approach to testing, deployment, and incident response.This talk walks through the system we built to do it safely and repeatedly: a vendor-agnostic qualification pipeline, an automated test framework with tag-driven selection for both routine releases and emergency response, progressive canary deployment gated by automated health checks, and a severity-calibrated emergency rollout process. We share two production war stories that shaped the pipeline, covering silent forwarding plane failures and a multi-vendor optics interoperability issue spanning three vendors on a single link.
Sujitha Sekar Rajan:
Sujitha Sekar Rajan is a Network Production Engineer at Meta, where she focuses on optimizing large-scale network infrastructure and ensuring operational reliability across one of the world's most complex production environments. Her work spans Network Design, OS Upgrades, BGP enhancements etc., with a consistent emphasis on measurable efficiency gains through cross-functional collaboration.
nanog.org/events/nanog-97/content/5754
Abstract
Large-scale network operators running multi-vendor environments face a recurring challenge: how to upgrade network operating systems across thousands of devices without compromising reliability. Upgrade too slowly and you accumulate security exposure, vendor end-of-life risk, and missing capabilities. Upgrade too fast and vendor regressions, configuration incompatibilities, and multi-vendor interaction failures can cause production incidents.
Therefore, managing network OS upgrades across tens and thousands of multi-vendor devices demands a disciplined approach to testing, deployment, and incident response.This talk walks through the system we built to do it safely and repeatedly: a vendor-agnostic qualification pipeline, an automated test framework with tag-driven selection for both routine releases and emergency response, progressive canary deployment gated by automated health checks, and a severity-calibrated emergency rollout process. We share two production war stories that shaped the pipeline, covering silent forwarding plane failures and a multi-vendor optics interoperability issue spanning three vendors on a single link.
Sujitha Sekar Rajan:
Sujitha Sekar Rajan is a Network Production Engineer at Meta, where she focuses on optimizing large-scale network infrastructure and ensuring operational reliability across one of the world's most complex production environments. Her work spans Network Design, OS Upgrades, BGP enhancements etc., with a consistent emphasis on measurable efficiency gains through cross-functional collaboration.
nanog.org/events/nanog-97/content/5754










