Hardware Podcasts

Death by Uptime

Oxide and Friends · 8 Dec 2025 · 1h 00m · Episode 34

What this episode covers

Bryan Cantrill, Adam Leventhal, Cliff Biffle, Matt Keeter, and Will Chandler of Oxide Computer Company discuss debugging a production failure where all 16 compute sled service processors (SPs) became unresponsive after ~188 days of uptime. The root cause was an unhandled interrupt from the STM32H7 Ethernet controller's management packet counter, which triggered an interrupt storm when the counter reached its halfway mark, starving the Hubris OS tasks of CPU time. The team used the Hubris debugger and serial wire debug interface to reproduce the issue by forcing the counter near overflow, confirming the hardware behavior described in the misleading STM32 datasheet. They implemented a fix in the Hubris firmware to disable the unmaskable interrupt, resolving the 'death by uptime' issue.

Most relevant to

large enterprises

Topics

Mentioned

Companies
Oxide Computer Company
Chips
STM32H7
Tools
MDB
Also
Hubris

Related episodes from other shows

Other shows on the same ground, closest first.

  • Embedded.fm · 1h 01m

    38: Blame the Monkey

    Elecia White and producer Chris White discuss strategies for debugging 'insurmountable' problems in embedded systems, ranging from race conditions and stack overflows to non-reproducible hardware issues. They advocate for methodical approaches such as drawing system diagrams, writing detailed emails to clarify thoughts, and using the scientific method to isolate variables. The conversation also covers the importance of logging, reverting to known good code, and knowing when to walk away or accept a bug as a feature.

    Debugging & failure analysisEmbedded firmware
    8 chapters

  • The Amp Hour · 57m

    Finding Hardware Flaws with Laura Abbott

    Laura Abbott, an engineer at Oxide Computer, discusses her accidental discovery of a buffer overflow vulnerability in the ROM update mechanism of the NXP LPC55S69 microcontroller while working on Oxide's server root of trust. She explains how Oxide is rebuilding the server from the ground up with a focus on hardware security, utilizing their open-source operating system Hubris and the Rust programming language to mitigate risks like memory safety issues. The conversation covers the architecture of the server's secure components, including the service processor and TrustZone-M, as well as the challenges of remote hardware debugging and contributing to the Linux kernel community.

    Security & privacyMicrocontrollers & dev platforms
    6 chapters

  • Embedded.fm · 52m

    467: Temporary Axolotl

    Christopher White discusses his viral moment involving a Chevy dealership's ChatGPT-powered chatbot, his struggles with contractor burnout, and the frustrating state of embedded development tools. He details specific technical failures with STM32CubeIDE, CMSIS drivers, and J-Link debuggers, highlighting the lack of standardization in the industry. The episode also covers personal updates, including his music with Flavigula, the Pass the Bricks charity, and Nelson Asinowski's donation of 8051 microcontrollers.

    EDA & CAD toolsDebugging & failure analysis
    7 chapters

  • Embedded.fm · 56m

    319: Squidly Tentacles

    Hosts Elecia White and Christopher White discuss the technical implications of the 2038 time bug, the utility of the 'single-file libraries' GitHub repository for embedded development, and Christopher's hands-on MIDI project using an STM32 NUCLEO-144 board. Elecia also shares her exploration of Tiny ML for PID controller tuning and her attempt to apply for a NASA NIAC grant using curved crease origami for sensor systems.

    Microcontrollers & dev platformsEmbedded firmware
    7 chapters

  • Embedded.fm · 1h 12m

    390: Irresponsible At the Time

    Tyler Hoffman, co-founder of Memfault, discusses the challenges of managing firmware updates, device diagnostics, and remote monitoring for millions of IoT devices. He draws on his experience at Pebble and Fitbit to explain why existing cloud platforms are often unsuitable for resource-constrained embedded systems, highlighting the need for secure boot, staged rollouts, and efficient heartbeat reporting. Hoffman details how Memfault’s platform helps engineers detect crashes, track battery life trends, and prioritize bugs by aggregating data from large device fleets. The conversation emphasizes the importance of defensive programming and building robust OTA systems to prevent bricking devices at scale.

    Embedded firmwareConnected products & IoT
    8 chapters

The summary, chapters and topics are generated from the episode’s audio by this directory, not written by Oxide and Friends. The episode itself belongs to its publisher.