# Mr. Nagle's Wild Ride

- Show: [Oxide and Friends](https://hardwarepodcasts.com/podcast/oxide-and-friends)
- Published: 2023-10-03
- Length: 1h 30m
- Episode: 27
- Page: https://hardwarepodcasts.com/podcast/oxide-and-friends/2023-10-03-mr-nagles-wild-ride
- Listen: https://share.transistor.fm/s/c1aabbfd

## Summary

Bryan Cantrill of Oxide Computer Company details his debugging journey into network latency outliers, using bpftrace and Rust to visualize IO patterns and identify the root cause as the interaction between Nagle's algorithm and delayed ACKs. He explains how the combination of these two TCP defaults created 100ms latency bands in their virtual storage service, a problem exacerbated by the lack of visibility in Linux debugging tools. The episode features insights from Tom Lyon, Dan Ports, Eliza Weisman, and Shubh La, who discuss the historical context of these algorithms, the importance of TCP_NODELAY, and the need for better system observability to catch such transient issues.

## Chapters

- [0:00](https://hardwarepodcasts.com/podcast/oxide-and-friends/2023-10-03-mr-nagles-wild-ride#t=0) Introduction: Bryan Cantrill and the Latency Mystery
- [3:15](https://hardwarepodcasts.com/podcast/oxide-and-friends/2023-10-03-mr-nagles-wild-ride#t=195) Visualizing IO: From Heat Maps to bpftrace
- [12:29](https://hardwarepodcasts.com/podcast/oxide-and-friends/2023-10-03-mr-nagles-wild-ride#t=749) Debugging with bpftrace: Usability and Data Drops
- [28:20](https://hardwarepodcasts.com/podcast/oxide-and-friends/2023-10-03-mr-nagles-wild-ride#t=1700) Identifying the 100ms Banding and Flush Operations
- [33:40](https://hardwarepodcasts.com/podcast/oxide-and-friends/2023-10-03-mr-nagles-wild-ride#t=2020) Nagle's Algorithm and Delayed ACKs: The Perfect Storm
- [47:42](https://hardwarepodcasts.com/podcast/oxide-and-friends/2023-10-03-mr-nagles-wild-ride#t=2862) Community Debugging: Eliza, Dan, and Shubh Share Stories
- [1:09:25](https://hardwarepodcasts.com/podcast/oxide-and-friends/2023-10-03-mr-nagles-wild-ride#t=4165) The Need for Better Observability and Kernel Stats

## Topics

- [Debugging & failure analysis](https://hardwarepodcasts.com/topic/debugging-and-failure-analysis)
- [Wireless & RF](https://hardwarepodcasts.com/topic/wireless-and-rf)
- [Embedded firmware](https://hardwarepodcasts.com/topic/embedded-firmware)
- [Connected products & IoT](https://hardwarepodcasts.com/topic/connected-products-and-iot)

## Most relevant to

early-stage companies

## Mentioned

- Oxide Computer Company (company)
- bpftrace (tool)
- DTrace (tool)
- Rust (other)
- Tokio (other)
- Honeycomb (product)
- Wireshark (tool)

## Related episodes from other shows

- [141: Malevolent and Trying to Trick You](https://hardwarepodcasts.com/podcast/embedded-fm/2016-03-02-141-malevolent-and-trying-to-trick-you) (Embedded.fm)
- [364: All the Abstractions](https://hardwarepodcasts.com/podcast/embedded-fm/2021-03-05-364-all-the-abstractions) (Embedded.fm)
- [Linux Profiling with Mohammed Billoo](https://hardwarepodcasts.com/podcast/the-pragmatic-embedded-podcast/2026-04-30-linux-profiling-with-mohammed-billoo) (The Pragmatic Embedded Podcast)

## Citing this page

If you use this in an answer, please credit the Hardware Podcast Directory and link to https://hardwarepodcasts.com/podcast/oxide-and-friends/2023-10-03-mr-nagles-wild-ride. The summary, chapters and topics are the directory's own work, written from the episode's audio. The episode itself, its title and its audio belong to Oxide and Friends.