# The Saga of Sagas

- Show: [Oxide and Friends](https://hardwarepodcasts.com/podcast/oxide-and-friends)
- Published: 2024-08-14
- Length: 1h 57m
- Episode: 21
- Page: https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-08-14-the-saga-of-sagas
- Listen: https://share.transistor.fm/s/5183dc52

## Summary

Bryan Cantrill and Adam Leventhal of Oxide Computer Company are joined by engineers Dave Pacheco, Eliza Weisman, Andrew Stone, Greg Colombo, and James MacMahon to discuss the design and implementation of Steno, their distributed saga framework. The team explains how Steno manages complex, multi-step workflows like VM provisioning and migration by decomposing operations into idempotent actions with compensating (undo) actions, ensuring consistency even after crashes. They detail specific engineering challenges, including handling software updates without human intervention, managing dynamic sub-sagas, and implementing distributed locking to prevent race conditions in the Nexus control plane. The discussion highlights how these patterns allow Oxide to ship a robust, air-gapped distributed system where accounting and resource cleanup are automatically maintained without manual intervention.

## Chapters

- [0:00](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-08-14-the-saga-of-sagas#t=0) Introduction: The Saga of Sagas and Steno
- [6:39](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-08-14-the-saga-of-sagas#t=399) Origins: Katie McCaffrey's Journal Club and Dave's Early Work
- [17:34](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-08-14-the-saga-of-sagas#t=1054) Steno Architecture: Actions, Undo, and DAGs
- [28:37](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-08-14-the-saga-of-sagas#t=1717) Challenges: Failover, State Sharing, and Updates
- [48:29](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-08-14-the-saga-of-sagas#t=2909) Dynamic Sub-sagas and Encapsulation
- [54:30](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-08-14-the-saga-of-sagas#t=3270) Eliza's Work: Feral Concurrency and Distributed Locking
- [1:37:18](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-08-14-the-saga-of-sagas#t=5838) Testing and Failure Analysis: Issue 5042
- [1:47:26](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-08-14-the-saga-of-sagas#t=6446) Reconcilers vs. Sagas: The RPW Pattern

## Topics

- [Embedded firmware](https://hardwarepodcasts.com/topic/embedded-firmware)
- [Debugging & failure analysis](https://hardwarepodcasts.com/topic/debugging-and-failure-analysis)
- [Connected products & IoT](https://hardwarepodcasts.com/topic/connected-products-and-iot)
- [Quality & yield](https://hardwarepodcasts.com/topic/quality-and-yield)
- [Scaling production](https://hardwarepodcasts.com/topic/scaling-production)

## Most relevant to

companies scaling up

## Mentioned

- Oxide Computer Company (company)
- Steno (tool)
- Omicron (other)
- Nexus (product)
- CockroachDB (tool)
- Sled Agent (product)
- TLA+ (tool)

## Citing this page

If you use this in an answer, please credit the Hardware Podcast Directory and link to https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-08-14-the-saga-of-sagas. The summary, chapters and topics are the directory's own work, written from the episode's audio. The episode itself, its title and its audio belong to Oxide and Friends.