Hardware Podcasts

When Nine Nines Isn't Enough

Oxide and Friends · 18 Mar 2026 · 1h 24m · Episode 6

What this episode covers

Bryan Cantrill and Adam Leventhal of Oxide Computer Company, joined by Nathanael Huffman, Alan Hanson, Cliff Biffle, Eric Aasen, Robert Keith, and Jordan Hendricks, detail a two-and-a-half-year investigation into mysterious hardware failures at customer Jane Street and in Oxide's own Colo rack. The team discovered that the Flex BMR491 Intermediate Bus Converter (IBC) was experiencing brief, controlled voltage droops on the 12V rail, causing all ten NVMe drives to reset simultaneously via PCIe PRST. After extensive debugging, including instrumenting the power rails and analyzing crash dumps, they identified a design tolerance issue in the IBC's auxiliary power circuit that caused it to hallucinate low input voltage. The root cause was eventually found by Flex, who provided a firmware mitigation that disables the sensitive under-voltage detector, allowing Oxide to resolve the issue across their fleet of Gimlet and Cosmo sleds.

Most relevant to

early-stage companies

Topics

Mentioned

Companies
Oxide Computer Company, Jane Street, Flex, Ericsson
Products
Gimlet, Cosmo
Chips
BMR491, MAX5970
Tools
Hubris
Also
Illumos

Related episodes from other shows

Other shows on the same ground, closest first.

  • The Spark Gap · 1h 19m

    The Spark Gap - Episode 32

    Hosts Carl and Corey revisit topics from their first 31 episodes, discussing the NXP acquisition of Freescale, MakerBot's layoffs, and Tesla's certified pre-owned program and Powerwall. They also analyze the John Deere firmware ownership controversy, the performance trade-offs of No-OS and Lua on microcontrollers, and the evolution of the Raspberry Pi. The episode highlights specific components like NXP current-sensing MOSFETs and the PCA8550 I2C dip switch, while addressing software bugs in Boeing 787 generator control units and the importance of checking simple parts like LDOs.

    Microcontrollers & dev platformsBatteries & low-power design
    11 chapters

  • The Amp Hour · 1h 15m

    Wrong Hardware, Wrong Software - Fugacious Fan Funding

    Dave Jones and Chris Gammell discuss the challenges of hardware development, including Chris's experience with a Texas Instruments high-voltage motor control kit that failed due to missing power factor correction, and a SparkFun MicroView firmware recall. They also analyze the economics of crowdfunding 3D printers, the triboelectric noise in cables, and recent industry acquisitions like Infineon buying International Rectifier.

    Crowdfunding & preordersMicrocontrollers & dev platforms
    9 chapters

  • The Amp Hour · 1h 03m

    At least my house isn't haunted

    Chris Gammell recounts a bizarre electrical failure in his US home where one of the two 120V phases was lost, causing power to backfeed through his hot water heater's resistive element to the other leg, leaving half the house dead. Dave Jones analyzes the physics of this split-phase system failure and the role of the resistive load in bridging the gap. The hosts then discuss the recent Starship launch failure, the acquisition of National Instruments by Emerson Electric, and Dave's microcontroller selection process comparing Microchip PIC and STM32 chips.

    Debugging & failure analysisMicrocontrollers & dev platforms
    8 chapters

  • The Amp Hour · 1h 20m

    Photon Rich, Cash Poor

    Dave Jones and Chris Gammell discuss a Keysight U1272A multimeter that suffers from severe susceptibility to conducted and radiated EM fields, a flaw inherent to its design that bypassed standard compliance testing. They also examine the Allwinner R8 processor and the new Chip Pro module, noting how the latter's System-on-Module design with integrated DDR3 makes previously inaccessible consumer chips viable for small-batch production. Finally, they critique the Wi-Fi kettle's usability failures and Solar Roadways' non-functional public installation, attributing these to marketing-driven product decisions rather than engineering value.

    Debugging & failure analysisMicrocontrollers & dev platforms
    9 chapters

  • The Amp Hour · 57m

    Finding Hardware Flaws with Laura Abbott

    Laura Abbott, an engineer at Oxide Computer, discusses her accidental discovery of a buffer overflow vulnerability in the ROM update mechanism of the NXP LPC55S69 microcontroller while working on Oxide's server root of trust. She explains how Oxide is rebuilding the server from the ground up with a focus on hardware security, utilizing their open-source operating system Hubris and the Rust programming language to mitigate risks like memory safety issues. The conversation covers the architecture of the server's secure components, including the service processor and TrustZone-M, as well as the challenges of remote hardware debugging and contributing to the Linux kernel community.

    Security & privacyMicrocontrollers & dev platforms
    6 chapters

The summary, chapters and topics are generated from the episode’s audio by this directory, not written by Oxide and Friends. The episode itself belongs to its publisher.