Hardware Podcasts

Debugging Methodologies

Oxide and Friends · 17 May 2022 · 1h 30m · Episode 14

What this episode covers

Jordan Hendricks and Luqman Aden of Oxide Computer Company detail their methodologies for debugging complex hardware and compiler issues. Hendricks explains how he used the KMDB in-situ debugger to diagnose an NVMe reset hang, discovering that switching from abrupt to normal shutdown notifications resolved the issue. Aden describes his process for debugging a rustc segfault involving inline assembly in async blocks, using LLVM IR analysis and Rust Analyzer to isolate a bug in the compiler's code generation. The episode also covers a third case involving a triple fault caused by undefined behavior in a bare-metal bootloader, highlighting the importance of tooling and collaborative debugging culture at Oxide.

Most relevant to

early-stage companies

Topics

Mentioned

Companies
Oxide Computer Company
Products
NanoBlurs
Tools
KMDB, rustc, LLVM, Rust Analyzer, MIRI
Also
NVMe

Related episodes from other shows

Other shows on the same ground, closest first.

  • Embedded.fm · 1h 01m

    38: Blame the Monkey

    Elecia White and producer Chris White discuss strategies for debugging 'insurmountable' problems in embedded systems, ranging from race conditions and stack overflows to non-reproducible hardware issues. They advocate for methodical approaches such as drawing system diagrams, writing detailed emails to clarify thoughts, and using the scientific method to isolate variables. The conversation also covers the importance of logging, reverting to known good code, and knowing when to walk away or accept a bug as a feature.

    Debugging & failure analysisEmbedded firmware
    8 chapters

  • The Amp Hour · 57m

    Finding Hardware Flaws with Laura Abbott

    Laura Abbott, an engineer at Oxide Computer, discusses her accidental discovery of a buffer overflow vulnerability in the ROM update mechanism of the NXP LPC55S69 microcontroller while working on Oxide's server root of trust. She explains how Oxide is rebuilding the server from the ground up with a focus on hardware security, utilizing their open-source operating system Hubris and the Rust programming language to mitigate risks like memory safety issues. The conversation covers the architecture of the server's secure components, including the service processor and TrustZone-M, as well as the challenges of remote hardware debugging and contributing to the Linux kernel community.

    Security & privacyMicrocontrollers & dev platforms
    6 chapters

  • Embedded.fm · 1h 09m

    334: Bag of Heuristics

    Matt Godbolt, creator of Compiler Explorer (godbolt.org) and the jsbeeb BBC Micro emulator, discusses the evolution of his open-source tools, the technical details of compiler internals, and the nostalgia of 6502 computing. He explains how Compiler Explorer helps developers understand assembly output, manage ABI constraints, and debug compiler bugs, while also detailing the security and privacy challenges of running arbitrary code on a public web service. Godbolt also shares insights into the BBC Micro's hardware quirks, the impact of Hyrum's Law on legacy systems, and the shift from transparent hardware to opaque modern processors.

    EDA & CAD toolsEmbedded firmware
    7 chapters

  • Embedded.fm · 56m

    435: Sad Lack of Gnomes

    Hosts Elecia White and Chris White discuss recent shifts in the embedded development ecosystem, specifically the shutdown of the Mbed compiler ecosystem and its replacement by Keil Studio Cloud, which they find significantly more complex. They also explore new developer tools like GitHub Codespaces for cloud-based coding and Linode for hosting self-managed Mastodon servers. The conversation covers technical nuances of C language quirks, such as compound literals and flexible array members, and the mechanics of hardware versus software breakpoints in debugging.

    EDA & CAD toolsEmbedded firmware
    8 chapters

  • Embedded.fm · 1h 12m

    215: Heisenbugs

    Guest Alvaro Prieto joins hosts Alicia White and Christopher White to discuss the fundamentals of debugging embedded systems, distinguishing between software debuggers, hardware programmers, and microprocessor debug support. They cover the evolution from print-based debugging to using tools like GDB, LLDB, and Visual Studio Code, while explaining the roles of hardware interfaces like J-Tag, SWD, and Trace. The conversation highlights the trade-offs between open-source toolchains (GCC, OpenOCD) and paid professional IDEs (Keil, IAR, Code Composer Studio), emphasizing how paid tools simplify setup for complex chip architectures. Prieto also shares anecdotes about his personal projects, including a cheese-aging cave and a new reverse engineering podcast.

    Debugging & failure analysisEDA & CAD tools
    9 chapters

The summary, chapters and topics are generated from the episode’s audio by this directory, not written by Oxide and Friends. The episode itself belongs to its publisher.