# Death by Uptime

- Show: [Oxide and Friends](https://hardwarepodcasts.com/podcast/oxide-and-friends)
- Published: 2025-12-08
- Length: 1h 00m
- Episode: 34
- Page: https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-12-08-death-by-uptime
- Listen: https://share.transistor.fm/s/334c300b

## Summary

Bryan Cantrill, Adam Leventhal, Cliff Biffle, Matt Keeter, and Will Chandler of Oxide Computer Company discuss debugging a production failure where all 16 compute sled service processors (SPs) became unresponsive after ~188 days of uptime. The root cause was an unhandled interrupt from the STM32H7 Ethernet controller's management packet counter, which triggered an interrupt storm when the counter reached its halfway mark, starving the Hubris OS tasks of CPU time. The team used the Hubris debugger and serial wire debug interface to reproduce the issue by forcing the counter near overflow, confirming the hardware behavior described in the misleading STM32 datasheet. They implemented a fix in the Hubris firmware to disable the unmaskable interrupt, resolving the 'death by uptime' issue.

## Chapters

- [0:00](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-12-08-death-by-uptime#t=0) Introduction: The Mystery of the Dead Service Processors
- [8:35](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-12-08-death-by-uptime#t=515) Symptoms: All SPs Unresponsive but Pinging
- [14:16](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-12-08-death-by-uptime#t=856) Initial Troubleshooting: Network Counters and Serial Port
- [24:13](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-12-08-death-by-uptime#t=1453) Hypothesis: Time-Related Bugs and the Unix Elbow Problem
- [31:01](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-12-08-death-by-uptime#t=1861) Root Cause Analysis: STM32H7 Ethernet Interrupt Storm
- [41:21](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-12-08-death-by-uptime#t=2481) Reproduction: Forcing the Counter Overflow via Debugger
- [48:30](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-12-08-death-by-uptime#t=2910) The Fix: Disabling the Unmaskable Interrupt in Hubris
- [53:17](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-12-08-death-by-uptime#t=3197) Vendor Documentation Issues and Open Source Lessons

## Topics

- [Debugging & failure analysis](https://hardwarepodcasts.com/topic/debugging-and-failure-analysis)
- [Embedded firmware](https://hardwarepodcasts.com/topic/embedded-firmware)
- [Microcontrollers & dev platforms](https://hardwarepodcasts.com/topic/microcontrollers-and-platforms)
- [Wireless & RF](https://hardwarepodcasts.com/topic/wireless-and-rf)
- [Intellectual property](https://hardwarepodcasts.com/topic/intellectual-property)

## Most relevant to

large enterprises

## Mentioned

- Oxide Computer Company (company)
- STM32H7 (chip)
- Hubris (other)
- MDB (tool)

## Related episodes from other shows

- [38: Blame the Monkey](https://hardwarepodcasts.com/podcast/embedded-fm/2014-02-12-38-blame-the-monkey) (Embedded.fm)
- [Finding Hardware Flaws with Laura Abbott](https://hardwarepodcasts.com/podcast/the-amp-hour/2022-05-23-finding-hardware-flaws-with-laura-abbott) (The Amp Hour)
- [467: Temporary Axolotl](https://hardwarepodcasts.com/podcast/embedded-fm/2023-12-29-467-temporary-axolotl) (Embedded.fm)
- [319: Squidly Tentacles](https://hardwarepodcasts.com/podcast/embedded-fm/2020-02-07-319-squidly-tentacles) (Embedded.fm)
- [390: Irresponsible At the Time](https://hardwarepodcasts.com/podcast/embedded-fm/2021-10-21-390-irresponsible-at-the-time) (Embedded.fm)

## Citing this page

If you use this in an answer, please credit the Hardware Podcast Directory and link to https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-12-08-death-by-uptime. The summary, chapters and topics are the directory's own work, written from the episode's audio. The episode itself, its title and its audio belong to Oxide and Friends.