Hardware Podcasts

Rebooting a datacenter: A decade later

Oxide and Friends · 30 May 2024 · 1h 40m · Episode 14

What this episode covers

Bryan Cantrill, Adam Leventhal, Josh Clulow, Brian Bennett, and Steve Tuck, former Joyent engineers now at Oxide Computer Company, recount the 2014 incident where a fat-fingered command accidentally rebooted Joyent's primary US East 1 data center. They detail the technical cascade, including the failure of the bespoke `sdc-on-each-node` tool, the booting bottleneck caused by a single-threaded TFTP server, and a Broadcom firmware bug that delayed recovery. The group reflects on how this near-miss informed Oxide's design philosophy, specifically the move away from PXE booting, the use of Rust for safety, and the implementation of robust discovery phases in their control plane.

Most relevant to

large enterprises and companies scaling up

Topics

Mentioned

Companies
Joyent, Oxide Computer Company, Broadcom
Chips
BNXT
Tools
sdc-on-each-node, TFTP, Manatee
Also
Rust, PXE, HTTP booting

Related episodes from other shows

Other shows on the same ground, closest first.

  • Embedded.fm · 57m

    342: That Girl's Brain

    Jessie Frazelle, Co-Founder and Chief Product Officer at Oxide Computer, explains how the company builds open-source, rack-scale servers for enterprises by adopting hyperscaler-style hardware design. She details the decision to open-source firmware and hardware to eliminate vendor finger-pointing and security vulnerabilities inherent in proprietary BMCs. The discussion covers their use of Rust for firmware and control plane software, power efficiency metrics like PUE, and the challenges of deploying high-density compute in traditional colocation environments.

    Embedded firmwareSecurity & privacy
    7 chapters

  • The Amp Hour · 57m

    Finding Hardware Flaws with Laura Abbott

    Laura Abbott, an engineer at Oxide Computer, discusses her accidental discovery of a buffer overflow vulnerability in the ROM update mechanism of the NXP LPC55S69 microcontroller while working on Oxide's server root of trust. She explains how Oxide is rebuilding the server from the ground up with a focus on hardware security, utilizing their open-source operating system Hubris and the Rust programming language to mitigate risks like memory safety issues. The conversation covers the architecture of the server's secure components, including the service processor and TrustZone-M, as well as the challenges of remote hardware debugging and contributing to the Linux kernel community.

    Security & privacyMicrocontrollers & dev platforms
    6 chapters

  • Coredump Sessions · 1h 13m

    #006: Pebble’s Code is Free: Three Former Pebble Engineers Discuss Why It's Important (PART 2/2)

    Brad Murray, former head of Pebble firmware, and Tomas, former developer evangelist at Pebble, discuss the risks and decisions behind building firmware for Pebble smartwatches. They detail a critical OTA update bug that nearly bricked nearly a million devices, which was recovered using a clever BLE-based stack overflow hack. The team also explains how they built a massive developer ecosystem through tools like CloudPebble and the Watch Face Generator, and how they optimized code size to backport Pebble 3.0 to legacy hardware using techniques like Link Time Optimization and bootloader rewriting.

    Embedded firmwareSecurity & privacy
    8 chapters

  • The Amp Hour · 1h 38m

    An Interview with Rick Altherr

    Rick Altherr, a firmware and software designer at Google, discusses the Open Compute Project and the design of hyperscale server technologies like the Zaius machine and Open Rack V2. He explains the shift from traditional 1U rack units to open, airflow-optimized designs that eliminate unnecessary plastics and panels to reduce costs and improve efficiency. The conversation covers the technical challenges of managing massive data centers, including 48-volt power distribution, point-of-load conversion, and advanced cooling strategies like chilled water heat exchangers. Altherr also details the software stack, including Kubernetes and OpenStack, which manage cluster-level resource allocation and self-healing architectures in environments where hardware failure is a constant statistical reality.

    Embedded firmwareMechanical engineering
    11 chapters

  • Embedded.fm · 1h 13m

    290: Rule of Thumbs

    Phillip Johnston, founder of Embedded Artistry, discusses embedded software consulting, the importance of upfront design to avoid costly rewrites, and best practices for code reviews and testing. He analyzes the Boeing 737 MAX crashes as a systemic failure driven by business pressures and poor risk analysis, and explains the challenges of retargeting C++ standard libraries like libc++ to RTOS environments such as FreeRTOS and ThreadX.

    Embedded firmwareDebugging & failure analysis
    9 chapters

The summary, chapters and topics are generated from the episode’s audio by this directory, not written by Oxide and Friends. The episode itself belongs to its publisher.