Hardware Podcasts

Adventures in Data Corruption

Oxide and Friends · 10 Jul 2025 · 1h 43m · Episode 23

What this episode covers

Bryan Cantrill, Adam Leventhal, John Gallagher, and Rain Paharia of Oxide Computer Company discuss a bizarre data corruption bug discovered two years prior during their sled recovery process. The team traced the issue to CPU speculative execution across kernel page table isolation boundaries, specifically involving a 'stowaway' mapping in the kernel-only CR3 that leaked into the TLB. They detail the debugging odyssey, from initial hash mismatches in the Installinator tool to identifying the 'haunted virtual address' region and finally confirming the microarchitectural speculation bug with AMD.

Most relevant to

companies scaling up

Topics

Mentioned

Companies
Oxide Computer Company, AMD
Products
Installinator, Wicket, ZFS, Omicron
Chips
Milan, Genoa
Tools
Tokyo, DTrace

Related episodes from other shows

Other shows on the same ground, closest first.

  • The Amp Hour · 57m

    Finding Hardware Flaws with Laura Abbott

    Laura Abbott, an engineer at Oxide Computer, discusses her accidental discovery of a buffer overflow vulnerability in the ROM update mechanism of the NXP LPC55S69 microcontroller while working on Oxide's server root of trust. She explains how Oxide is rebuilding the server from the ground up with a focus on hardware security, utilizing their open-source operating system Hubris and the Rust programming language to mitigate risks like memory safety issues. The conversation covers the architecture of the server's secure components, including the service processor and TrustZone-M, as well as the challenges of remote hardware debugging and contributing to the Linux kernel community.

    Security & privacyMicrocontrollers & dev platforms
    6 chapters

  • Embedded.fm · 58m

    414: Puff, the Magically Secure Dragon

    Laura Abbott, a firmware engineer at Oxide Computing, discusses two silicon bugs found in the NXP LPC55S69 microcontroller, including a buffer overflow in the ROM that allows cloning the device's PUF identity. She explains how she used Ghidra to reverse-engineer the undocumented ROM code and details the security implications for TrustZone and Secure Boot. The conversation also covers Oxide's approach to hiring, their use of Rust for firmware, and the challenges of working with proprietary silicon.

    Debugging & failure analysisMicrocontrollers & dev platforms
    7 chapters

  • Unnamed Reverse Engineering Podcast · 1h 01m

    055 - Stacks Of Bricked Chips

    Laura Abbott, a software developer at Oxide Computer, explains how her team reverse engineered the NXP LPC55S69 microcontroller to build a root of trust for their servers. She details discovering an undocumented ROM patching hardware block that allowed privilege escalation, leading to a CVE and a DEF CON presentation. Abbott also discusses her background in Linux kernel development, the transition to Rust for microcontrollers, and the challenges of working with contradictory chip documentation.

    Chips, FPGAs & semiconductorsSecurity & privacy
    7 chapters

  • Embedded.fm · 1h 01m

    38: Blame the Monkey

    Elecia White and producer Chris White discuss strategies for debugging 'insurmountable' problems in embedded systems, ranging from race conditions and stack overflows to non-reproducible hardware issues. They advocate for methodical approaches such as drawing system diagrams, writing detailed emails to clarify thoughts, and using the scientific method to isolate variables. The conversation also covers the importance of logging, reverting to known good code, and knowing when to walk away or accept a bug as a feature.

    Debugging & failure analysisEmbedded firmware
    8 chapters

  • Embedded.fm · 1h 21m

    229: Slinky with a Lot of Math

    Nick Kartsioukas, an information security engineer known as ExplodingLemur, discusses the practical application of threat modeling, the mechanics of Spectre and Meltdown vulnerabilities, and essential user-level security hygiene. He explains how speculative execution and cache timing attacks allow user-space processes to leak kernel memory, highlighting the difficulty of patching legacy industrial and medical systems. For general users, he recommends using password managers, two-factor authentication, and ad blockers, while emphasizing the importance of data backups and mental health resources.

    Security & privacyMicrocontrollers & dev platforms
    8 chapters

The summary, chapters and topics are generated from the episode’s audio by this directory, not written by Oxide and Friends. The episode itself belongs to its publisher.