I Know This! (Purpose-built systems with general-purpose guts)

Oxide and Friends · 8 Feb 2022 · 1h 20m · Episode 4

What this episode covers

Bryan Cantrill and Adam Leventhal of Oxide Computer Company discuss the complexities of hardware support, debugging, and customer relations, drawing on their past experiences at Sun Microsystems and Delphix. They recount specific incidents, including a fatal e-cache parity error on SPARC systems, a kernel modification that corrupted ZFS data structures, and the implementation of strict support protocols like the 'three strikes' rule. The conversation highlights the tension between customer autonomy and system integrity, emphasizing the importance of transparency, auditing, and post-incident reviews in both on-premise and SaaS environments.

Mentioned

Companies
Oxide Computer Company, Sun Microsystems, DelphX, Dell, NXP
Products
ZFS, LPC55
Chips
SPARC
Tools
mdb
Also
MITRE, CVE

Other shows on the same ground, closest first.

  • The Amp Hour · 57m

    Finding Hardware Flaws with Laura Abbott

    Laura Abbott, an engineer at Oxide Computer, discusses her accidental discovery of a buffer overflow vulnerability in the ROM update mechanism of the NXP LPC55S69 microcontroller while working on Oxide's server root of trust. She explains how Oxide is rebuilding the server from the ground up with a focus on hardware security, utilizing their open-source operating system Hubris and the Rust programming language to mitigate risks like memory safety issues. The conversation covers the architecture of the server's secure components, including the service processor and TrustZone-M, as well as the challenges of remote hardware debugging and contributing to the Linux kernel community.

  • Embedded.fm · 58m

    141: Malevolent and Trying to Trick You

    Julia Evans, a software engineer at Stripe, discusses using Linux system-level tools like `strace`, `perf`, and `top` to debug applications as black boxes. She explains how `strace` reveals system calls to understand program behavior without reading source code, and how `perf top` identifies CPU-intensive functions like garbage collection. Evans also shares her philosophy on debugging, emphasizing the importance of maintaining confidence and understanding that computers are logical machines, even when bugs seem impossible.

  • The Amp Hour · 1h 05m

    Occam's Engineering Razor

    Dave Jones and Chris Gammell discuss the pitfalls of over-complicating troubleshooting, illustrated by Dave's solar panel saga where a tiny coax cable shadow caused a 20% power drop. They explore the psychology of engineering resilience, referencing Ben Krasnow's persistence, and debate the efficacy of project management methodologies like Gantt charts versus agile demos in hardware development. The conversation also covers the semiconductor industry, analyzing Intel's foundry ambitions and the challenges of automotive chip sourcing.

  • Embedded.fm · 1h 18m

    166: Sardine Tornado

    Bob Apthorpe, a nuclear engineer with a background in web development, discusses the intersection of safety engineering, software reliability, and improv comedy. He explains how lessons from severe accident analysis, such as hydrogen control and molten fuel debris simulation, apply to software design, particularly regarding warning systems and common-cause failures. Apthorpe argues that the 'noise' of excessive compiler warnings or system alerts mirrors the 'annunciator panel' problem at Three Mile Island, advocating for rigorous signal-to-noise management. He also shares how stage improv helped him navigate workplace dynamics and accept failure as a necessary part of learning in hardware and software projects.

  • The Amp Hour · 1h 38m

    An Interview with Rick Altherr

    Rick Altherr, a firmware and software designer at Google, discusses the Open Compute Project and the design of hyperscale server technologies like the Zaius machine and Open Rack V2. He explains the shift from traditional 1U rack units to open, airflow-optimized designs that eliminate unnecessary plastics and panels to reduce costs and improve efficiency. The conversation covers the technical challenges of managing massive data centers, including 48-volt power distribution, point-of-load conversion, and advanced cooling strategies like chilled water heat exchangers. Altherr also details the software stack, including Kubernetes and OpenStack, which manage cluster-level resource allocation and self-healing architectures in environments where hardware failure is a constant statistical reality.

The summary, chapters and topics are generated from the episode’s audio by this directory, not written by Oxide and Friends. The episode itself belongs to its publisher.