# A Debugging Odyssey

- Show: [Oxide and Friends](https://hardwarepodcasts.com/podcast/oxide-and-friends)
- Published: 2022-12-19
- Length: 1h 35m
- Episode: 38
- Page: https://hardwarepodcasts.com/podcast/oxide-and-friends/2022-12-19-a-debugging-odyssey
- Listen: https://share.transistor.fm/s/8488f6b6

## Summary

Dave Pacheco of Oxide Computer Company details his months-long debugging odyssey to resolve a critical memory corruption bug in the Go runtime on Illumos. He explains how he used DTrace, MDB, and binary patching to identify that the OS failed to preserve YMM registers during signal handling, a flaw that could have caused catastrophic data corruption in their CockroachDB control plane. The episode highlights the importance of rigorous documentation, deep tooling mastery, and the cultural value of tackling seemingly unreproducible low-level failures.

## Chapters

- [0:00](https://hardwarepodcasts.com/podcast/oxide-and-friends/2022-12-19-a-debugging-odyssey#t=0) Introduction: Dave Pacheco and the Flaky Test
- [1:05](https://hardwarepodcasts.com/podcast/oxide-and-friends/2022-12-19-a-debugging-odyssey#t=65) The Debugging Odyssey Begins: Reproducing Go Runtime Panics
- [5:20](https://hardwarepodcasts.com/podcast/oxide-and-friends/2022-12-19-a-debugging-odyssey#t=320) Blind Alleys: libumem, AMD Errata, and Memory Corruption Hypotheses
- [13:39](https://hardwarepodcasts.com/podcast/oxide-and-friends/2022-12-19-a-debugging-odyssey#t=819) Shifting Tactics: Learning the Go Memory Allocator and GC
- [27:03](https://hardwarepodcasts.com/podcast/oxide-and-friends/2022-12-19-a-debugging-odyssey#t=1623) Tooling Dividends: MDB Type Definitions and DTrace Instrumentation
- [51:15](https://hardwarepodcasts.com/podcast/oxide-and-friends/2022-12-19-a-debugging-odyssey#t=3075) The Breakthrough: YMM Register Preservation Failure
- [1:09:29](https://hardwarepodcasts.com/podcast/oxide-and-friends/2022-12-19-a-debugging-odyssey#t=4169) Verification and Impact: Binary Patching and Production Risks
- [1:13:44](https://hardwarepodcasts.com/podcast/oxide-and-friends/2022-12-19-a-debugging-odyssey#t=4424) Broader Context: Signal Handling Complexity and Future Hardware
- [1:26:26](https://hardwarepodcasts.com/podcast/oxide-and-friends/2022-12-19-a-debugging-odyssey#t=5186) Lessons Learned: Documentation, Culture, and Persistence

## Topics

- [Debugging & failure analysis](https://hardwarepodcasts.com/topic/debugging-and-failure-analysis)
- [Embedded firmware](https://hardwarepodcasts.com/topic/embedded-firmware)
- [AI & machine learning](https://hardwarepodcasts.com/topic/ai-and-machine-learning)

## Most relevant to

companies scaling up

## Mentioned

- Oxide Computer Company (company)
- CockroachDB (product)
- Illumos (other)
- MDB (tool)
- DTrace (tool)
- Go (other)

## Related episodes from other shows

- [38: Blame the Monkey](https://hardwarepodcasts.com/podcast/embedded-fm/2014-02-12-38-blame-the-monkey) (Embedded.fm)
- [Finding Hardware Flaws with Laura Abbott](https://hardwarepodcasts.com/podcast/the-amp-hour/2022-05-23-finding-hardware-flaws-with-laura-abbott) (The Amp Hour)
- [141: Malevolent and Trying to Trick You](https://hardwarepodcasts.com/podcast/embedded-fm/2016-03-02-141-malevolent-and-trying-to-trick-you) (Embedded.fm)
- [175: How Hard Could It Be?](https://hardwarepodcasts.com/podcast/embedded-fm/2016-11-09-175-how-hard-could-it-be) (Embedded.fm)
- [Super Tecmo Bug](https://hardwarepodcasts.com/podcast/the-amp-hour/2023-02-13-super-tecmo-bug) (The Amp Hour)

## Citing this page

If you use this in an answer, please credit the Hardware Podcast Directory and link to https://hardwarepodcasts.com/podcast/oxide-and-friends/2022-12-19-a-debugging-odyssey. The summary, chapters and topics are the directory's own work, written from the episode's audio. The episode itself, its title and its audio belong to Oxide and Friends.