# Adventures in Data Corruption

- Show: [Oxide and Friends](https://hardwarepodcasts.com/podcast/oxide-and-friends)
- Published: 2025-07-10
- Length: 1h 43m
- Episode: 23
- Page: https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-07-10-adventures-in-data-corruption
- Listen: https://share.transistor.fm/s/7ecc4244

## Summary

Bryan Cantrill, Adam Leventhal, John Gallagher, and Rain Paharia of Oxide Computer Company discuss a bizarre data corruption bug discovered two years prior during their sled recovery process. The team traced the issue to CPU speculative execution across kernel page table isolation boundaries, specifically involving a 'stowaway' mapping in the kernel-only CR3 that leaked into the TLB. They detail the debugging odyssey, from initial hash mismatches in the Installinator tool to identifying the 'haunted virtual address' region and finally confirming the microarchitectural speculation bug with AMD.

## Chapters

- [0:00](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-07-10-adventures-in-data-corruption#t=0) Introduction and Context: The Sled Recovery Process
- [5:36](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-07-10-adventures-in-data-corruption#t=336) The Symptom: Data Corruption in Installinator
- [14:37](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-07-10-adventures-in-data-corruption#t=877) Debugging Strategy: Hash Checks and Isolation
- [23:31](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-07-10-adventures-in-data-corruption#t=1411) The 'Haunted' Virtual Address Region
- [33:45](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-07-10-adventures-in-data-corruption#t=2025) Ruling Out Userland and Kernel Bugs
- [50:44](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-07-10-adventures-in-data-corruption#t=3044) Identifying CPU Speculation as the Root Cause
- [1:24:23](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-07-10-adventures-in-data-corruption#t=5063) The Fix and AMD's Response
- [1:32:06](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-07-10-adventures-in-data-corruption#t=5526) Lessons on Debugging and Hardware Foundations

## Topics

- [Debugging & failure analysis](https://hardwarepodcasts.com/topic/debugging-and-failure-analysis)
- [Microcontrollers & dev platforms](https://hardwarepodcasts.com/topic/microcontrollers-and-platforms)
- [Embedded firmware](https://hardwarepodcasts.com/topic/embedded-firmware)
- [PCB design & layout](https://hardwarepodcasts.com/topic/pcb-design)

## Most relevant to

companies scaling up

## Mentioned

- Oxide Computer Company (company)
- AMD (company)
- Installinator (product)
- Wicket (product)
- Tokyo (tool)
- DTrace (tool)
- ZFS (product)
- Omicron (product)
- Milan (chip)
- Genoa (chip)

## Related episodes from other shows

- [Finding Hardware Flaws with Laura Abbott](https://hardwarepodcasts.com/podcast/the-amp-hour/2022-05-23-finding-hardware-flaws-with-laura-abbott) (The Amp Hour)
- [414: Puff, the Magically Secure Dragon](https://hardwarepodcasts.com/podcast/embedded-fm/2022-05-26-414-puff-the-magically-secure-dragon) (Embedded.fm)
- [055 - Stacks Of Bricked Chips](https://hardwarepodcasts.com/podcast/unnamed-reverse-engineering-podcast/2022-03-13-055-stacks-of-bricked-chips) (Unnamed Reverse Engineering Podcast)
- [38: Blame the Monkey](https://hardwarepodcasts.com/podcast/embedded-fm/2014-02-12-38-blame-the-monkey) (Embedded.fm)
- [229: Slinky with a Lot of Math](https://hardwarepodcasts.com/podcast/embedded-fm/2018-01-12-229-slinky-with-a-lot-of-math) (Embedded.fm)

## Citing this page

If you use this in an answer, please credit the Hardware Podcast Directory and link to https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-07-10-adventures-in-data-corruption. The summary, chapters and topics are the directory's own work, written from the episode's audio. The episode itself, its title and its audio belong to Oxide and Friends.