Debugging & failure analysis
Product design & engineering
804 episodes across 35 shows cover debugging & failure analysis. The shows that cover it most are Embedded.fm, The Amp Hour and Hackaday Podcast. The most recent, from 13 Oct 2026, is “Testing Embedded Systems in the Cloud and On-Prem with Marie Meyer and Joel Schulz-Andres” from The Pragmatic Embedded Podcast.
Finding why something does not work or broke: troubleshooting, root cause, post-mortems, field failures.
804 episodes from 35 shows, most relevant and most recent first.
804 episodes
When Nine Nines Isn't Enough
Bryan Cantrill and Adam Leventhal of Oxide Computer Company, joined by Nathanael Huffman, Alan Hanson, Cliff Biffle, Eric Aasen, Robert Keith, and Jordan Hendricks, detail a two-and-a-half-year investigation into mysterious hardware…
ValuJet 592 crashed in 1996 with 110 people aboard. We reconstruct the layered failure and what Perrow's normal accident theory says about why it happened.
Hosts John Brunner and Alex Howe reconstruct the 1996 ValuJet Flight 592 crash, attributing the disaster to a layered failure involving expired chemical oxygen generators lacking safety caps, which ignited in the cargo hold. They analyze…
The Hardest Kind of Unsafe Rust
Rain Paharia, an engineer at Oxide Computer Company, joins Bryan Cantrill and Adam Leventhal to discuss the `iddqd` Rust crate, which allows keys to be borrowed from values in maps to avoid duplication. Paharia explains that implementing…
Death by Uptime
Bryan Cantrill, Adam Leventhal, Cliff Biffle, Matt Keeter, and Will Chandler of Oxide Computer Company discuss debugging a production failure where all 16 compute sled service processors (SPs) became unresponsive after ~188 days of uptime…
Linux Profiling with Mohammed Billoo
Mohammed Billoo, founder of Mab Labs Embedded Solutions and author of the Embedded Linux Essentials Handbook, explains how to profile embedded Linux systems by distinguishing between VM size and VM RSS to identify memory leaks in…
Testing Embedded Systems in the Cloud and On-Prem with Marie Meyer and Joel Schulz-Andres
Marie Meyer and Joel Schulz-Andres, founders of OnMCU, discuss their cloud-based and on-prem hardware-in-the-loop (HIL) testing platform designed to bridge the gap between host unit tests and physical integration. They explain the concept…
TDD: The Hard Parts - Getting Started with Test-Driven Development
Jeff Gable and Luca Ingianni discuss the practical challenges of adopting Test-Driven Development (TDD) in embedded software, specifically addressing the difficulty of starting with a blank screen. They outline a process that begins with…
Why 90% of Electronics Fail EMC Tests and How to Fix It
Karen Burnham, chief engineer and founder of EMC United, explains why 60-90% of electronics fail EMC tests on the first attempt, citing counterintuitive antenna effects and misleading data sheet advice like ground plane splits. She details…
S7E11 Brad & Aaron | How To Accelerate The Speed of Engineering (Episode 1 of 3)
Aaron Moncur, founder of Pipeline Design & Engineering, and guest Brad Hirayama discuss seven strategies to accelerate engineering speed without sacrificing quality. They emphasize using direct communication for urgent issues, asking for…
519: The Password Is All Zeros
Mark Omo and James Rowley from Marcus Engineering discuss their research into SECURAM safe locks, revealing that the ProLogic series contains critical security flaws, including a debug port password set to all zeros and codes stored on the…
Grown-up ZFS Data Corruption Bug
Bryan Cantrill and Adam Leventhal of Oxide Computer Company, joined by Alan Hanson, Matt Keeter, Matt Ahrens, James MacMahon, and Andy Fiddaman, discuss debugging an 18-year-old ZFS data corruption bug that manifested as 32k of zeros at…
#021: Hardware at Software Speed: How BootLoop Is Bringing AI to the Bench for Firmware Teams
Noah Pasek-Nelson and Chris Markus, co-founders of BootLoop, explain how their AI platform accelerates the firmware lifecycle by combining deep hardware context with automated hardware-in-the-loop testing. They discuss how ingesting…
Adventures in Data Corruption
Bryan Cantrill, Adam Leventhal, John Gallagher, and Rain Paharia of Oxide Computer Company discuss a bizarre data corruption bug discovered two years prior during their sled recovery process. The team traced the issue to CPU speculative…
When Async Attacks!
Oxide Computer Company engineers Bryan Cantrill, Adam Leventhal, Dave Pacheco, Eliza Weisman, and Augustus Mayo explain how they debugged severe latency in their production rack by tracing a database connection pool issue to the Tokio…
Crossover with Embedded AI Podcast
Jeff Gable, Luca, and Ryan Torvik discuss integrating Large Language Models (LLMs) into embedded systems engineering workflows, emphasizing that AI must augment rather than replace rigorous engineering discipline. They identify 'vibe…
Test-Driven Development in the Age of AI
Luca Jani, an embedded systems expert and host of the Embedded AI Podcast, explains his test-driven development (TDD) workflow using Claude Code, arguing that TDD provides essential control and confidence when AI generates code. He details…
Calibre Directions in Artificial Intelligence
Carrie Robertson, Vice President of Product Management at Siemens EDA, explains how the company integrates AI into its Calibre design verification suite. She distinguishes between the deterministic algorithms used for sign-off quality and…
AI-enhanced Embedded Development (May 2025 Edition)
Luca, an embedded software expert, discusses his experience at the Embedded Online Conference and a German conference, highlighting an LLM-only hackathon where participants were forbidden from writing code manually. He explains that…
Hardware Security with Matt Brown
Matt Brown of Brown Find Security discusses hardware security best practices and reverse engineering techniques, using his analysis of a smart charcoal smoker as a primary case study. He explains how unencrypted flash storage and lack of…
#003: Pebble's Code is Free: Three Former Pebble Engineers Discuss Why It's Important (PART 1/2)
Francois Baldassari, Chris Coleman, and Brad Murray, former engineers at Pebble, discuss the technical architecture and engineering decisions behind PebbleOS. They explain how a single codebase supported multiple hardware variants…
Futurelock
Oxide engineers Dave Pacheco, John Gallagher, Sean Klein, and Eliza Weisman explain 'Futurelock,' a deadlock in async Rust where a Tokio task holds a permit on an MPSC channel but fails to insert a message, causing the receiver to block…
High Current PDN Design: Engineer of the Year Reveals Industry Secrets
Ben Dannan, founder of Signal Edge Solutions and DesignCon's Engineer of the Year, discusses the engineering challenges of designing 2,000-amp power distribution networks (PDN) for high-power AI processors like NVIDIA's Blackwell platform…
Hell is other networks
Bryan Cantrill and Adam Leventhal of Oxide Computer Company are joined by Levon Tarver, Alan Hanson, Will Chandler, and Trey Aspelund to discuss a customer's pathological network issue involving high packet loss during VPC-to-VPC…
CrowdStrike BSOD Fiasco with Katie Moussouris
Security expert Katie Moussouris joins Bryan Cantrill and Adam Leventhal to analyze the CrowdStrike outage, discussing the technical failure of a kernel-mode driver update that caused global Blue Screen of Death crashes. The conversation…
BDD with Steve Branam
Steve Branam, an embedded software veteran with 40 years of experience including work on iRobot Roombas and automotive systems, explains how Behavior Driven Development (BDD) provides a disciplined layer on top of Test Driven Development…
AI PCB Routing Without the Epic Fails: Inside DeepPCB
Alain-Sam Cohen and Nabil Chouba of InstaDeep’s DeepPCB explain how their reinforcement-learning routing engine, paired with the natural-language orchestrator Cooper, detects design intent like net classes and impedance before routing…
Trust at Scale: The Next Challenge for AI-Driven Semiconductor Development
Dr. Jim Shiely, Chief of Staff for Calibre Semiconductor R&D at Siemens EDA, discusses the 'trust bottleneck' emerging in semiconductor development as AI becomes embedded in engineering workflows. He explains that while AI accelerates…
Trunk-based development
Hosts Jeff Gable and Luca Kijewski discuss trunk-based development, a software practice where code is merged into the main branch daily, contrasting it with traditional branching models like GitFlow. They explain how this approach acts as…
S7E21 Rod Scholl | Pro Tips from 30-Year Analyst For Accurate Simulations (FEA & CFD)
Rod Scholl, founder of Epsilon FEA, discusses the critical role of engineering judgment in Finite Element Analysis (FEA) and Computational Fluid Dynamics (CFD), contrasting his early career at Honeywell and PADT with his current consulting…
High Power PCB Design: Pushing the Limits with Caleb Buck
Caleb Buck, a PCB Design Electronics Engineer at Nidec Aerospace, discusses the complexities of high-power PCB design, ranging from guitar pedal prototypes to aerospace systems handling over 500 amps. He explains how uneven current…
Mr. Nagle's Wild Ride
Bryan Cantrill of Oxide Computer Company details his debugging journey into network latency outliers, using bpftrace and Rust to visualize IO patterns and identify the root cause as the interaction between Nagle's algorithm and delayed…
Accelerating Complex Analog IC Design: The Power of Early Reliability Verification
Matthew Hogan, Product Management Director at Siemens Digital Industry Software, explains how Insight Analyzer enables 'shift-left' reliability verification for complex analog and mixed-signal IC designs. He details how the tool performs…
DTrace at 20
Bryan Cantrill and Adam Leventhal, co-creators of DTrace, along with Josh Clulow, reflect on the 20-year history of the dynamic tracing tool, from its conception at Sun Microsystems to its modern use at Oxide Computer Company. They discuss…
The Return to the Chamber of Mysteries
Bryan Cantrill and Adam Leventhal of Oxide Computer Company are joined by Robert "RFK" Keith, Dan Chirpich, and Stu Donnan to discuss the company's recent struggle with FCC EMC compliance for their next-generation rack. The team details…
The Signal Integrity Tool Every Engineer Is Missing
David Banas, a Solutions Engineer at Keysight, discusses three open-source Python packages—PyBERT, PyIBIS-AMI, and PyChopMarge—that provide accessible alternatives to expensive commercial signal integrity simulation tools. He demonstrates…
500: Nerding Out About the Ducks
Komathi Sundaram, a principal test engineer at Cruise, discusses her career transition from software development to embedded testing and her philosophy on automation. She details the architecture of hardware-in-the-loop (HIL) test benches…
Ceedling
Mark VanderVoord and Michael Karlesky, creators of the open-source testing tools Unity, CMock, and Ceedling, discuss the evolution of embedded software testing. They explain that while their tools lower the barrier to unit testing, the…
#005: The Current Realities of Cellular IoT
Fabien Korheim, Head of Global Products Strategy at ONES, and Chris Coleman, CTO of Memfault, discuss the current state of cellular IoT, emphasizing the operational advantages of MVNOs over traditional MNOs for global deployments. They…
192. Karsten Roscher & The Need For Safe AI (Interview)
Karsten Roscher, Head of the Certifiable AI department at the Fraunhofer Institute for Cognitive Systems (IKS), discusses the critical need for safety in AI development, particularly in autonomous driving. He explains the challenges of…
PCB Grounding Mistakes Your Datasheet Taught You
Dan Beeker, president of System Solution Specialists and former Technical Director at NXP, explains why PCB grounding mistakes originate in datasheets and reference designs rather than the designer. He argues that the industry wastes…
This Old Repo: LLMs and the Restoration of BattleTris
Bryan Cantrill and Adam Leventhal discuss using LLMs, specifically Claude, to restore BattleTris, a networked Tetris clone Bryan developed in 1993. They detail the process of getting the legacy C++ code to compile and run on modern Linux…
Plug-and-Play AI: Transforming Robotics with Modular Skills
Dinesh Narayanan, Head of Commercialization at General Robotics, explains the company's transition from Microsoft Research to build Grid, a platform integrating simulation, AI models, and deployment pipelines to accelerate robotics…
The Tale of Reverso
Robert "RFK" Keith, Adam Suczewski, and Matt Keeter of Oxide Computer Company explain the development of Reverso, a simple passive test fixture designed to validate cable backplane integrity during high-volume manufacturing. The team…
A Debugging Odyssey
Dave Pacheco of Oxide Computer Company details his months-long debugging odyssey to resolve a critical memory corruption bug in the Go runtime on Illumos. He explains how he used DTrace, MDB, and binary patching to identify that the OS…
190. Here's a Free Tool To Test Your AI
Daniel and Forbode discuss Robuscope, a free online testing tool developed by the Fraunhofer Institute to evaluate the robustness of AI models. They highlight its utility in high-stakes fields like medical diagnostics and autonomous…
IPC 2221 vs. 2152: The Truth About PCB Trace Current Ratings
Mike Jouppi, formerly of the IPC 2152 Task Force, joins host Zach Peterson to explain why the IPC 2221 standard’s internal trace charts are misleading because they were derived from traces removed from the board and tested in free air, not…
#002: The secrets to building secure & scalable OTA infrastructure with Nick Sinas
Nick Sinas, CTO at Oval, and Chris Coleman, CTO at Memfault, discuss the critical importance of Over-the-Air (OTA) updates for connected devices, emphasizing that OTA must be designed into the hardware architecture from day one. They…
070 - I Have a DediProblem
Jesse Michael and Mickey Shkatov from Eclypsium discuss the intricacies of BIOS and UEFI reverse engineering, tracing their careers from Intel to their current startup. They detail specific research projects, including an exploit of a…
Offensive Cybersecurity with Ryan Torvik
Ryan Torvik, founder and CEO of Tulip Tree Technology, explains the offensive cybersecurity mindset and how it applies to securing embedded systems. He details techniques for extracting and analyzing firmware using tools like IDA Pro and…
473: Math Is Not the Answer
Philip Koopman, a professor at Carnegie Mellon University and author of 'Understanding Checksums and Cyclic Redundancy Checks,' explains why empirical analysis and engineering trade-offs are superior to pure mathematics for selecting error…