# Rebooting a datacenter: A decade later

- Show: [Oxide and Friends](https://hardwarepodcasts.com/podcast/oxide-and-friends)
- Published: 2024-05-30
- Length: 1h 40m
- Episode: 14
- Page: https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-05-30-rebooting-a-datacenter-a-decade-later
- Listen: https://share.transistor.fm/s/81324a6a

## Summary

Bryan Cantrill, Adam Leventhal, Josh Clulow, Brian Bennett, and Steve Tuck, former Joyent engineers now at Oxide Computer Company, recount the 2014 incident where a fat-fingered command accidentally rebooted Joyent's primary US East 1 data center. They detail the technical cascade, including the failure of the bespoke `sdc-on-each-node` tool, the booting bottleneck caused by a single-threaded TFTP server, and a Broadcom firmware bug that delayed recovery. The group reflects on how this near-miss informed Oxide's design philosophy, specifically the move away from PXE booting, the use of Rust for safety, and the implementation of robust discovery phases in their control plane.

## Chapters

- [0:00](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-05-30-rebooting-a-datacenter-a-decade-later#t=0) Introduction and Context of the 2014 Outage
- [6:27](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-05-30-rebooting-a-datacenter-a-decade-later#t=387) The Fat-Fingered Command and sdc-on-each-node Failure
- [31:06](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-05-30-rebooting-a-datacenter-a-decade-later#t=1866) Boot Bottlenecks: TFTP, Manatee, and CPU Contention
- [38:14](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-05-30-rebooting-a-datacenter-a-decade-later#t=2294) The Broadcom BNXT Firmware Bug and Manual Recovery
- [45:28](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-05-30-rebooting-a-datacenter-a-decade-later#t=2728) Customer Communication and Crisis Management
- [1:08:18](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-05-30-rebooting-a-datacenter-a-decade-later#t=4098) Lessons Learned: Safety, Rust, and Oxide's Architecture

## Topics

- [Debugging & failure analysis](https://hardwarepodcasts.com/topic/debugging-and-failure-analysis)
- [Embedded firmware](https://hardwarepodcasts.com/topic/embedded-firmware)
- [Leadership, teams & culture](https://hardwarepodcasts.com/topic/leadership-and-culture)
- [AI & machine learning](https://hardwarepodcasts.com/topic/ai-and-machine-learning)
- [Careers & education](https://hardwarepodcasts.com/topic/engineering-careers-and-education)

## Most relevant to

large enterprises, companies scaling up

## Mentioned

- Joyent (company)
- Oxide Computer Company (company)
- sdc-on-each-node (tool)
- TFTP (tool)
- Manatee (tool)
- Broadcom (company)
- BNXT (chip)
- Rust (other)
- PXE (other)
- HTTP booting (other)

## Related episodes from other shows

- [342: That Girl's Brain](https://hardwarepodcasts.com/podcast/embedded-fm/2020-08-27-342-that-girls-brain) (Embedded.fm)
- [Finding Hardware Flaws with Laura Abbott](https://hardwarepodcasts.com/podcast/the-amp-hour/2022-05-23-finding-hardware-flaws-with-laura-abbott) (The Amp Hour)
- [#006: Pebble’s Code is Free: Three Former Pebble Engineers Discuss Why It's Important (PART 2/2)](https://hardwarepodcasts.com/podcast/coredump-sessions/2025-04-15-006-pebbles-code-is-free-three-former-pebble-engineers-discuss-why-its-important) (Coredump Sessions)
- [An Interview with Rick Altherr](https://hardwarepodcasts.com/podcast/the-amp-hour/2017-08-29-an-interview-with-rick-altherr) (The Amp Hour)
- [290: Rule of Thumbs](https://hardwarepodcasts.com/podcast/embedded-fm/2019-06-06-290-rule-of-thumbs) (Embedded.fm)

## Citing this page

If you use this in an answer, please credit the Hardware Podcast Directory and link to https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-05-30-rebooting-a-datacenter-a-decade-later. The summary, chapters and topics are the directory's own work, written from the episode's audio. The episode itself, its title and its audio belong to Oxide and Friends.