# When Nine Nines Isn't Enough

- Show: [Oxide and Friends](https://hardwarepodcasts.com/podcast/oxide-and-friends)
- Published: 2026-03-18
- Length: 1h 24m
- Episode: 6
- Page: https://hardwarepodcasts.com/podcast/oxide-and-friends/2026-03-18-when-nine-nines-isnt-enough
- Listen: https://share.transistor.fm/s/b401f7ad

## Summary

Bryan Cantrill and Adam Leventhal of Oxide Computer Company, joined by Nathanael Huffman, Alan Hanson, Cliff Biffle, Eric Aasen, Robert Keith, and Jordan Hendricks, detail a two-and-a-half-year investigation into mysterious hardware failures at customer Jane Street and in Oxide's own Colo rack. The team discovered that the Flex BMR491 Intermediate Bus Converter (IBC) was experiencing brief, controlled voltage droops on the 12V rail, causing all ten NVMe drives to reset simultaneously via PCIe PRST. After extensive debugging, including instrumenting the power rails and analyzing crash dumps, they identified a design tolerance issue in the IBC's auxiliary power circuit that caused it to hallucinate low input voltage. The root cause was eventually found by Flex, who provided a firmware mitigation that disables the sensitive under-voltage detector, allowing Oxide to resolve the issue across their fleet of Gimlet and Cosmo sleds.

## Chapters

- [0:00](https://hardwarepodcasts.com/podcast/oxide-and-friends/2026-03-18-when-nine-nines-isnt-enough#t=0) Introduction: Oxide Team Roll Call and Context
- [4:51](https://hardwarepodcasts.com/podcast/oxide-and-friends/2026-03-18-when-nine-nines-isnt-enough#t=291) Initial Failure at Jane Street: Sled 10 and ZFS Panics
- [12:31](https://hardwarepodcasts.com/podcast/oxide-and-friends/2026-03-18-when-nine-nines-isnt-enough#t=751) Debugging PCIe Resets and the MAX5970 Hot-Swap Controller
- [27:33](https://hardwarepodcasts.com/podcast/oxide-and-friends/2026-03-18-when-nine-nines-isnt-enough#t=1653) Instrumenting the 12V Rail and Discovering Voltage Droops
- [43:51](https://hardwarepodcasts.com/podcast/oxide-and-friends/2026-03-18-when-nine-nines-isnt-enough#t=2631) Engaging Flex and Identifying the BMR491 IBC
- [56:44](https://hardwarepodcasts.com/podcast/oxide-and-friends/2026-03-18-when-nine-nines-isnt-enough#t=3404) Software Mitigation: Handling Rapid Hot-Plug Events in Hubris
- [1:01:37](https://hardwarepodcasts.com/podcast/oxide-and-friends/2026-03-18-when-nine-nines-isnt-enough#t=3697) Root Cause: IBC Auxiliary Power Circuit Tolerance Issues
- [1:04:43](https://hardwarepodcasts.com/podcast/oxide-and-friends/2026-03-18-when-nine-nines-isnt-enough#t=3883) Implementing RFD 630 and Field Mitigation

## Topics

- [Debugging & failure analysis](https://hardwarepodcasts.com/topic/debugging-and-failure-analysis)
- [PCB design & layout](https://hardwarepodcasts.com/topic/pcb-design)
- [Batteries & low-power design](https://hardwarepodcasts.com/topic/batteries-and-low-power)
- [Embedded firmware](https://hardwarepodcasts.com/topic/embedded-firmware)
- [Manufacturing partners & offshoring](https://hardwarepodcasts.com/topic/manufacturing-partners)

## Most relevant to

early-stage companies

## Mentioned

- Oxide Computer Company (company)
- Jane Street (company)
- Flex (company)
- Ericsson (company)
- BMR491 (chip)
- MAX5970 (chip)
- Gimlet (product)
- Cosmo (product)
- Hubris (tool)
- Illumos (other)

## Related episodes from other shows

- [The Spark Gap - Episode 32](https://hardwarepodcasts.com/podcast/the-spark-gap/2015-05-06-the-spark-gap-episode-32) (The Spark Gap)
- [Wrong Hardware, Wrong Software - Fugacious Fan Funding](https://hardwarepodcasts.com/podcast/the-amp-hour/2014-09-08-wrong-hardware-wrong-software-fugacious-fan-funding) (The Amp Hour)
- [At least my house isn't haunted](https://hardwarepodcasts.com/podcast/the-amp-hour/2023-04-24-at-least-my-house-isnt-haunted) (The Amp Hour)
- [Photon Rich, Cash Poor](https://hardwarepodcasts.com/podcast/the-amp-hour/2016-10-14-photon-rich-cash-poor) (The Amp Hour)
- [Finding Hardware Flaws with Laura Abbott](https://hardwarepodcasts.com/podcast/the-amp-hour/2022-05-23-finding-hardware-flaws-with-laura-abbott) (The Amp Hour)

## Citing this page

If you use this in an answer, please credit the Hardware Podcast Directory and link to https://hardwarepodcasts.com/podcast/oxide-and-friends/2026-03-18-when-nine-nines-isnt-enough. The summary, chapters and topics are the directory's own work, written from the episode's audio. The episode itself, its title and its audio belong to Oxide and Friends.