Hardware Podcasts

Debugger-Driven Development

Oxide and Friends · 16 Jun 2025 · 1h 34m · Episode 20

What this episode covers

Bryan Cantrill and Adam Leventhal host Oxide colleagues Dave Pacheco, John Gallagher, Alan Hanson, and Eliza Weisman to discuss debugger-driven development at Oxide Computer Company. They detail the creation of omdb, a domain-specific debugging tool that allows developers to inspect and modify internal state across components like Nexus, CockroachDB, and Crucible. The team explains how omdb accelerated the development of complex systems like Reconfigurator and Fault Management by providing immediate, visual feedback and enabling incremental deployment of incomplete features. The episode highlights how this approach reduced risk, improved demo quality, and allowed the team to ship robust, rack-scale hardware and software faster.

Most relevant to

companies scaling up

Topics

Mentioned

Companies
Oxide Computer Company, Joyent
Products
Nexus, CockroachDB, Crucible, Reconfigurator, Solaris, Manta, Cosmo, Oximeter, Sled Agent, MGS
Tools
omdb, MDB, Humility

Related episodes from other shows

Other shows on the same ground, closest first.

  • Embedded.fm · 53m

    209: Debuggerception

    Pierre-Marie de Rodat, a software engineer at AdaCore, explains the inner workings of debuggers, focusing on how GDB uses DWARF debug information to interpret program state. He details how compilers generate this metadata to allow debuggers to handle optimizations and complex types, and discusses the use of Python scripting in GDB for custom pretty printers. The conversation also covers code coverage tools like GNATcoverage and the AdaCore-sponsored Make with Ada competition.

    Debugging & failure analysisEmbedded firmware
    6 chapters

  • Embedded.fm · 1h 17m

    395: I Can No Longer Play Ping Pong

    Tyler Hoffman, co-founder of Memfault, explains how his team at Pebble and later Fitbit transformed firmware development by standardizing developer environments, optimizing build systems, and automating debugging. He details the shift from slow, sequential IAR builds to parallel GCC builds using Make and Conda, which reduced build times from 18 minutes to 2 minutes. Hoffman also describes building web applications and GDB Python scripts to enable remote debugging of field crashes without physical debuggers. The discussion highlights the importance of dedicated developer productivity roles and the evolution of tools from one-off scripts to robust, company-wide platforms.

    Embedded firmwareDebugging & failure analysis
    8 chapters

  • Embedded.fm · 1h 01m

    489: Constructive Cat

    Elecia White and Christopher White discuss strategies for solo developers to conduct effective code reviews, such as using Git branches, PR templates, and automated static analysis to gain distance from their own code. They then provide a detailed technical explanation of GDB (GNU Debugger), covering the compilation process, debug symbols, and the architecture of cross-debugging via GDB servers and hardware programmers like J-Link and ST-Link. The episode also features Elecia White discussing her origami art installation at the Aptos Public Library and her Python-based pattern generation tools.

    Embedded firmwareDebugging & failure analysis
    5 chapters

  • Embedded.fm · 1h 38m

    200: Oops

    Hosts Alicia and Christopher White celebrate the 200th episode of Embedded.fm with a panel of former guests: Alvaro Prieto, Andrei Chichak, Elizabeth Brenner, Chris Svec, and Chris Gammell. The group discusses their favorite embedded tools, including Saleae logic analyzers, Digilent Analog Discovery, Beyond Compare, and HP16C calculators, while sharing humorous field stories and interview anecdotes. They reflect on the joys of working in embedded systems, such as seeing products on retail shelves and the satisfaction of debugging complex hardware-software interactions.

    Embedded firmwareDebugging & failure analysis
    10 chapters

  • Embedded.fm · 52m

    218: Neutron Star of Dev Boards

    Dirk Akeman, partnership marketing manager at SEGGER, explains the architecture and evolution of J-Link debuggers, detailing how internal microcontrollers and FPGAs enable high-speed communication and features like ETM trace. He discusses the company's rigorous testing methodology, which involves maintaining a library of nearly 3,000 evaluation boards and using their own tools for internal development to ensure reliability. Akeman also highlights the shift from reactive debugging to proactive verification using tools like Ozone and SystemView, which allow engineers to profile code coverage and trace interrupts in real-time to prevent bugs before delivery.

    Debugging & failure analysisEmbedded firmware
    9 chapters

The summary, chapters and topics are generated from the episode’s audio by this directory, not written by Oxide and Friends. The episode itself belongs to its publisher.