Uploaded November 2026 | Updated November 2026, 31 minutes ago
On November 12th, Outlook, Teams and Copilot stopped working in offices around the world. Downdetector got about 130,000 reports for Outlook alone. Engineers restarted the servers, they came back healthy, and the moment users returned, they went down again. Why did every restart fail?
- Not a cyber attack, not a traffic spike: Microsoft found no sign of malicious activity, and the load was not caused by more traffic
- The trigger: a network change made during an earlier capacity upgrade. Servers began failing their automated health checks
- The load balancer's rule: fail the health check, leave the pool. Removed servers' work moved onto the servers that were left
- The villain: the process that sets up secure connections. A new TLS handshake needs heavy public-key math; an open connection runs on cheap symmetric encryption. Too many new connections on too few servers, and it was overwhelmed
- The twist: a restarted server looks perfect with no traffic. Open the gate, every offline app reconnects at once, each needing a fresh handshake, and it goes down again. That's a thundering herd
- The fix: revert the network change, bypass the health check on some servers, throttle traffic, rebuild some servers on an older configuration. Recovery still took more than a day
System Design Interview takeaways:
1. A health check can turn a small fault into a big outage, so cap how many servers it can remove at once
2. New TLS connections are expensive, so clients retry with backoff and random jitter, never all together
3. After an outage, bring traffic back slowly
Comment and tell me: when Outlook went down, what did your office do?
Based on Microsoft's public status updates; the final root-cause report is still to come.
kgptalkie.com
#shorts #Microsoft365 #MicrosoftOutage #OutlookDown #TeamsDown #SystemDesign #SystemDesignInterview #LoadBalancer #TLS #KGPTalkie
On November 12th, Outlook, Teams and Copilot stopped working in offices around the world. Downdetector got about 130,000 reports for Outlook alone. Engineers restarted the servers, they came back healthy, and the moment users returned, they went down again. Why did every restart fail?
- Not a cyber attack, not a traffic spike: Microsoft found no sign of malicious activity, and the load was not caused by more traffic
- The trigger: a network change made during an earlier capacity upgrade. Servers began failing their automated health checks
- The load balancer's rule: fail the health check, leave the pool. Removed servers' work moved onto the servers that were left
- The villain: the process that sets up secure connections. A new TLS handshake needs heavy public-key math; an open connection runs on cheap symmetric encryption. Too many new connections on too few servers, and it was overwhelmed
- The twist: a restarted server looks perfect with no traffic. Open the gate, every offline app reconnects at once, each needing a fresh handshake, and it goes down again. That's a thundering herd
- The fix: revert the network change, bypass the health check on some servers, throttle traffic, rebuild some servers on an older configuration. Recovery still took more than a day
System Design Interview takeaways:
1. A health check can turn a small fault into a big outage, so cap how many servers it can remove at once
2. New TLS connections are expensive, so clients retry with backoff and random jitter, never all together
3. After an outage, bring traffic back slowly
Comment and tell me: when Outlook went down, what did your office do?
Based on Microsoft's public status updates; the final root-cause report is still to come.
kgptalkie.com
#shorts #Microsoft365 #MicrosoftOutage #OutlookDown #TeamsDown #SystemDesign #SystemDesignInterview #LoadBalancer #TLS #KGPTalkie










