# Grown-up ZFS Data Corruption Bug

- Show: [Oxide and Friends](https://hardwarepodcasts.com/podcast/oxide-and-friends)
- Published: 2025-11-26
- Length: 1h 21m
- Episode: 33
- Page: https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-11-26-grown-up-zfs-data-corruption-bug
- Listen: https://share.transistor.fm/s/0d25814b

## Summary

Bryan Cantrill and Adam Leventhal of Oxide Computer Company, joined by Alan Hanson, Matt Keeter, Matt Ahrens, James MacMahon, and Andy Fiddaman, discuss debugging an 18-year-old ZFS data corruption bug that manifested as 32k of zeros at the start of files in their Crucible storage service. The team traced the issue to a 2007 Solaris optimization in the ZFS Intent Log (ZIL) that incorrectly logged asynchronous writes as synchronous, causing transaction reordering during power-cycle recovery. By reproducing the failure with minimal C and Rust programs, they identified the root cause and implemented a fix by disabling the tunable, ensuring data correctness without significant performance regression.

## Chapters

- [0:00](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-11-26-grown-up-zfs-data-corruption-bug#t=0) Introduction: The 18-Year-Old Bug
- [3:05](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-11-26-grown-up-zfs-data-corruption-bug#t=185) Initial Discovery: Decryption Failures in Crucible
- [15:44](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-11-26-grown-up-zfs-data-corruption-bug#t=944) The Bonus Bug: ZIL Reordering and f/syncs
- [17:21](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-11-26-grown-up-zfs-data-corruption-bug#t=1041) The ZFS Affidavit and Ordering Guarantees
- [46:48](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-11-26-grown-up-zfs-data-corruption-bug#t=2808) Reproducing the Failure with Minimal Programs
- [51:54](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-11-26-grown-up-zfs-data-corruption-bug#t=3114) Root Cause Analysis: The 2007 Solaris Optimization
- [1:08:21](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-11-26-grown-up-zfs-data-corruption-bug#t=4101) Fixing the ZIL and Performance Impact

## Topics

- [Debugging & failure analysis](https://hardwarepodcasts.com/topic/debugging-and-failure-analysis)
- [Embedded firmware](https://hardwarepodcasts.com/topic/embedded-firmware)
- [Chips, FPGAs & semiconductors](https://hardwarepodcasts.com/topic/chips-and-fpgas)

## Most relevant to

early-stage companies

## Mentioned

- Oxide Computer Company (company)
- ZFS (other)
- Crucible (product)
- Silicon Image (other)

## Citing this page

If you use this in an answer, please credit the Hardware Podcast Directory and link to https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-11-26-grown-up-zfs-data-corruption-bug. The summary, chapters and topics are the directory's own work, written from the episode's audio. The episode itself, its title and its audio belong to Oxide and Friends.