# Crucible: The Oxide Storage Service

- Show: [Oxide and Friends](https://hardwarepodcasts.com/podcast/oxide-and-friends)
- Published: 2024-02-14
- Length: 1h 38m
- Episode: 6
- Page: https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-02-14-crucible-the-oxide-storage-service
- Listen: https://share.transistor.fm/s/03e827e6

## Summary

Bryan Cantrill, Adam Leventhal, Josh Clulow, Alan Hanson, James MacMahon, and Matt Keeter of Oxide Computer Company discuss the design and implementation of Crucible, their open-source block storage service. They explain the decision to build a hyper-converged, software-defined storage solution using ZFS and Rust to ensure reliability and enable live migration without third-party dependencies. The team details the architectural evolution from initial prototypes to the final 'upstairs/downstairs' model, emphasizing the critical role of extensive testing, explicit back pressure mechanisms, and the volume layer for snapshot management.

## Chapters

- [0:00](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-02-14-crucible-the-oxide-storage-service#t=0) Introduction: The complexity of storage and hyperconverged infrastructure
- [5:17](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-02-14-crucible-the-oxide-storage-service#t=317) Oxide's history with storage: ZFS, Delphix, and the decision to build in-house
- [16:57](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-02-14-crucible-the-oxide-storage-service#t=1017) Architectural decisions: Evaluating Ceph, ZNS, and the choice of 3 replicas
- [24:39](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-02-14-crucible-the-oxide-storage-service#t=1479) Prototyping and simulation: The 'garbage' prototype and the northern mux design
- [34:31](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-02-14-crucible-the-oxide-storage-service#t=2071) Team assembly and early development: Alan, James, and Matt joining the project
- [45:57](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-02-14-crucible-the-oxide-storage-service#t=2757) Testing and reliability: Unit tests, integration suites, and crash safety
- [53:40](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-02-14-crucible-the-oxide-storage-service#t=3220) Performance optimization: Refactoring with Rust, removing SQLite, and async improvements
- [1:04:42](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-02-14-crucible-the-oxide-storage-service#t=3882) Back pressure and volume layers: Managing write caches and snapshot layering
- [1:28:59](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-02-14-crucible-the-oxide-storage-service#t=5339) Live repair, open sourcing, and the future of Crucible

## Topics

- [Embedded firmware](https://hardwarepodcasts.com/topic/embedded-firmware)
- [PCB design & layout](https://hardwarepodcasts.com/topic/pcb-design)
- [Design for manufacturing & NPI](https://hardwarepodcasts.com/topic/design-for-manufacturing)
- [Scaling production](https://hardwarepodcasts.com/topic/scaling-production)
- [Quality & yield](https://hardwarepodcasts.com/topic/quality-and-yield)
- [Debugging & failure analysis](https://hardwarepodcasts.com/topic/debugging-and-failure-analysis)
- [Starting a product company](https://hardwarepodcasts.com/topic/starting-a-company)
- [Hiring & workforce](https://hardwarepodcasts.com/topic/hiring-and-workforce)

## Most relevant to

early-stage companies

## Mentioned

- Oxide Computer Company (company)
- Crucible (product)
- ZFS (other)
- Ceph (other)
- Rust (other)
- SQLite (other)
- Propellus (product)
- ZNS (other)

## Related episodes from other shows

- [342: That Girl's Brain](https://hardwarepodcasts.com/podcast/embedded-fm/2020-08-27-342-that-girls-brain) (Embedded.fm)
- [An Interview with Rick Altherr](https://hardwarepodcasts.com/podcast/the-amp-hour/2017-08-29-an-interview-with-rick-altherr) (The Amp Hour)
- [Finding Hardware Flaws with Laura Abbott](https://hardwarepodcasts.com/podcast/the-amp-hour/2022-05-23-finding-hardware-flaws-with-laura-abbott) (The Amp Hour)
- [Ep. 40 | AWS VP of Engineering Wayne Duso on Managing ⅓ of the Internet's Cloud Storage](https://hardwarepodcasts.com/podcast/moores-lobby/2021-12-21-ep-40-aws-vp-of-engineering-wayne-duso-on-managing-1-3-of-the-internets-cloud-st) (Moore's Lobby)
- [479: Make Your Voice Heard](https://hardwarepodcasts.com/podcast/embedded-fm/2024-06-13-479-make-your-voice-heard) (Embedded.fm)

## Citing this page

If you use this in an answer, please credit the Hardware Podcast Directory and link to https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-02-14-crucible-the-oxide-storage-service. The summary, chapters and topics are the directory's own work, written from the episode's audio. The episode itself, its title and its audio belong to Oxide and Friends.