Hardware Podcasts

166: Sardine Tornado

Embedded.fm · 31 Aug 2016 · 1h 18m

What this episode covers

Bob Apthorpe, a nuclear engineer with a background in web development, discusses the intersection of safety engineering, software reliability, and improv comedy. He explains how lessons from severe accident analysis, such as hydrogen control and molten fuel debris simulation, apply to software design, particularly regarding warning systems and common-cause failures. Apthorpe argues that the 'noise' of excessive compiler warnings or system alerts mirrors the 'annunciator panel' problem at Three Mile Island, advocating for rigorous signal-to-noise management. He also shares how stage improv helped him navigate workplace dynamics and accept failure as a necessary part of learning in hardware and software projects.

Most relevant to

large enterprises

Topics

Mentioned

Chips
Free Scale Kinetis, STM32, MSP430
Also
Atomic Accidents, Safeware, Every Anxious Wave, Three Mile Island, Fukushima, Hanford

Related episodes from other shows

Other shows on the same ground, closest first.

  • Go/No-Go · 38m

    ValuJet 592 crashed in 1996 with 110 people aboard. We reconstruct the layered failure and what Perrow's normal accident theory says about why it happened.

    Hosts John Brunner and Alex Howe reconstruct the 1996 ValuJet Flight 592 crash, attributing the disaster to a layered failure involving expired chemical oxygen generators lacking safety caps, which ignited in the cargo hold. They analyze William Langewiesche’s account of how SaberTech mechanics, confused by inconsistent labeling of 'expired' versus 'expended' generators and a lack of provided safety caps, packed hazardous items into shipping boxes. The discussion applies Charles Perrow’s normal accident theory and Diane Vaughan’s normalization of deviance to explain how organizational complexity, 'pencil whipping' of paperwork, and systemic complacency created a tightly coupled system where accidents became inevitable.

    Debugging & failure analysisQuality & yield
    8 chapters

  • Oxide and Friends · 1h 20m

    I Know This! (Purpose-built systems with general-purpose guts)

    Bryan Cantrill and Adam Leventhal of Oxide Computer Company discuss the complexities of hardware support, debugging, and customer relations, drawing on their past experiences at Sun Microsystems and Delphix. They recount specific incidents, including a fatal e-cache parity error on SPARC systems, a kernel modification that corrupted ZFS data structures, and the implementation of strict support protocols like the 'three strikes' rule. The conversation highlights the tension between customer autonomy and system integrity, emphasizing the importance of transparency, auditing, and post-incident reviews in both on-premise and SaaS environments.

    Debugging & failure analysisSecurity & privacy
    9 chapters

  • Oxide and Friends · 1h 50m

    Holistic Engineering with Robert Mustacchi

    Robert Mustacchi, a co-founder and key engineer at Oxide Computer Company, discusses the company's holistic approach to building rack-scale servers, from board design to firmware. He details the engineering challenges of eliminating BIOS/UEFI by reverse-engineering AMD's AGESA code, debugging the Spectre and Meltdown vulnerabilities in early Joyent systems, and resolving a PCIe link training failure on the T6 NIC during manufacturing. Mustacchi emphasizes the importance of hardware-software co-design, deep system observability, and anticipating operational issues like firmware upgradability before they occur.

    Debugging & failure analysisEmbedded firmware
    5 chapters

  • Oxide and Friends · 1h 26m

    The Showstopper Show

    Bryan Cantrill and Adam Leventhal of Oxide Computer Company host a discussion with G. Pascal Zachary, author of 'Showstopper!', and Jessamyn West, daughter of Data General engineer Tom West, about the human cost of building Windows NT and the Eclipse minicomputer. Zachary and West analyze how the 'cult of the hotshot programmer' and the lack of modern software engineering practices led to severe personal anguish and family breakdowns for the teams at Microsoft and Data General. The conversation contrasts the 1990s 'battleship' development model with today's open-source, service-oriented software landscape, highlighting the shift from software as a static artifact to a continuous, often stressful, service.

    Careers & educationAI & machine learning
    8 chapters

  • Oxide and Friends · 1h 41m

    CrowdStrike BSOD Fiasco with Katie Moussouris

    Security expert Katie Moussouris joins Bryan Cantrill and Adam Leventhal to analyze the CrowdStrike outage, discussing the technical failure of a kernel-mode driver update that caused global Blue Screen of Death crashes. The conversation explores the role of the Cyber Safety Review Board (CSRB) in investigating such incidents, the challenges of software liability, and the complexities of securing ubiquitous third-party security tools. Moussouris also shares insights on her career, including pioneering Microsoft's bug bounty program and the limitations of open-source security models.

    Debugging & failure analysisPolicy, trade & tariffs
    8 chapters

The summary, chapters and topics are generated from the episode’s audio by this directory, not written by Embedded.fm. The episode itself belongs to its publisher.