Hardware Podcasts

A Debugging Odyssey

Oxide and Friends · 19 Dec 2022 · 1h 35m · Episode 38

What this episode covers

Dave Pacheco of Oxide Computer Company details his months-long debugging odyssey to resolve a critical memory corruption bug in the Go runtime on Illumos. He explains how he used DTrace, MDB, and binary patching to identify that the OS failed to preserve YMM registers during signal handling, a flaw that could have caused catastrophic data corruption in their CockroachDB control plane. The episode highlights the importance of rigorous documentation, deep tooling mastery, and the cultural value of tackling seemingly unreproducible low-level failures.

Most relevant to

companies scaling up

Topics

Mentioned

Companies
Oxide Computer Company
Products
CockroachDB
Tools
MDB, DTrace
Also
Illumos, Go

Related episodes from other shows

Other shows on the same ground, closest first.

  • Embedded.fm · 1h 01m

    38: Blame the Monkey

    Elecia White and producer Chris White discuss strategies for debugging 'insurmountable' problems in embedded systems, ranging from race conditions and stack overflows to non-reproducible hardware issues. They advocate for methodical approaches such as drawing system diagrams, writing detailed emails to clarify thoughts, and using the scientific method to isolate variables. The conversation also covers the importance of logging, reverting to known good code, and knowing when to walk away or accept a bug as a feature.

    Debugging & failure analysisEmbedded firmware
    8 chapters

  • The Amp Hour · 57m

    Finding Hardware Flaws with Laura Abbott

    Laura Abbott, an engineer at Oxide Computer, discusses her accidental discovery of a buffer overflow vulnerability in the ROM update mechanism of the NXP LPC55S69 microcontroller while working on Oxide's server root of trust. She explains how Oxide is rebuilding the server from the ground up with a focus on hardware security, utilizing their open-source operating system Hubris and the Rust programming language to mitigate risks like memory safety issues. The conversation covers the architecture of the server's secure components, including the service processor and TrustZone-M, as well as the challenges of remote hardware debugging and contributing to the Linux kernel community.

    Security & privacyMicrocontrollers & dev platforms
    6 chapters

  • Embedded.fm · 58m

    141: Malevolent and Trying to Trick You

    Julia Evans, a software engineer at Stripe, discusses using Linux system-level tools like `strace`, `perf`, and `top` to debug applications as black boxes. She explains how `strace` reveals system calls to understand program behavior without reading source code, and how `perf top` identifies CPU-intensive functions like garbage collection. Evans also shares her philosophy on debugging, emphasizing the importance of maintaining confidence and understanding that computers are logical machines, even when bugs seem impossible.

    Debugging & failure analysisTest & measurement
    7 chapters

  • Embedded.fm · 1h 15m

    175: How Hard Could It Be?

    Jean Labrosse, founder of Micrium (now part of Silicon Labs) and author of the uC/OS real-time operating system, explains the origins of his software, which began after his team encountered a bug in a commercial RTOS that the vendor refused to fix. He details the technical challenges of building a robust kernel, including handling race conditions, priority inversion, and stack overflows, and discusses the importance of clean code and certification for medical and avionics applications. Labrosse also advocates for using commercial RTOS support over free alternatives like FreeRTOS, arguing that the 'free puppy' analogy highlights the hidden costs of maintenance and debugging time.

    Embedded firmwareStarting a product company
    10 chapters

  • The Amp Hour · 1h 10m

    Super Tecmo Bug

    Dave Jones and Chris Gammell discuss finding bugs in high-end oscilloscopes from Rigol and Rohde & Schwarz, highlighting the complexity of modern embedded firmware. They analyze the market for cheap RISC-V microcontrollers, specifically comparing the RV32I10 to alternatives like the PUYA PY32, and discuss Google's canceled Coral development board. The conversation also covers the complexity of automotive wiring harnesses, the bankruptcy of Lightyear, and skepticism around carbon credits and ESG scores.

    Debugging & failure analysisMicrocontrollers & dev platforms
    6 chapters

The summary, chapters and topics are generated from the episode’s audio by this directory, not written by Oxide and Friends. The episode itself belongs to its publisher.