Grown-up ZFS Data Corruption Bug

What this episode covers

Bryan Cantrill and Adam Leventhal of Oxide Computer Company, joined by Alan Hanson, Matt Keeter, Matt Ahrens, James MacMahon, and Andy Fiddaman, discuss debugging an 18-year-old ZFS data corruption bug that manifested as 32k of zeros at the start of files in their Crucible storage service. The team traced the issue to a 2007 Solaris optimization in the ZFS Intent Log (ZIL) that incorrectly logged asynchronous writes as synchronous, causing transaction reordering during power-cycle recovery. By reproducing the failure with minimal C and Rust programs, they identified the root cause and implemented a fix by disabling the tunable, ensuring data correctness without significant performance regression.