The Saga of Sagas

What this episode covers

Bryan Cantrill and Adam Leventhal of Oxide Computer Company are joined by engineers Dave Pacheco, Eliza Weisman, Andrew Stone, Greg Colombo, and James MacMahon to discuss the design and implementation of Steno, their distributed saga framework. The team explains how Steno manages complex, multi-step workflows like VM provisioning and migration by decomposing operations into idempotent actions with compensating (undo) actions, ensuring consistency even after crashes. They detail specific engineering challenges, including handling software updates without human intervention, managing dynamic sub-sagas, and implementing distributed locking to prevent race conditions in the Nexus control plane. The discussion highlights how these patterns allow Oxide to ship a robust, air-gapped distributed system where accounting and resource cleanup are automatically maintained without manual intervention.