# When Async Attacks!

- Show: [Oxide and Friends](https://hardwarepodcasts.com/podcast/oxide-and-friends)
- Published: 2025-06-27
- Length: 1h 53m
- Episode: 22
- Page: https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-06-27-when-async-attacks
- Listen: https://share.transistor.fm/s/32ca22c9

## Summary

Oxide Computer Company engineers Bryan Cantrill, Adam Leventhal, Dave Pacheco, Eliza Weisman, and Augustus Mayo explain how they debugged severe latency in their production rack by tracing a database connection pool issue to the Tokio runtime's LIFO slot optimization. They utilized DTrace, USDT probes, and custom state maps to visualize thread and task activity, discovering that a compute-bound certificate processing task was blocking the async runtime from stealing work. The root cause was identified as the LIFO slot preventing work stealing, which they resolved by disabling the optimization and adding deeper runtime instrumentation.

## Chapters

- [0:00](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-06-27-when-async-attacks#t=0) Introduction: Audio issues and production context
- [6:52](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-06-27-when-async-attacks#t=412) Augustus Mayo reports latency and database claim errors
- [14:06](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-06-27-when-async-attacks#t=846) Using DieselDTrace and USDT to exonerate the database
- [22:38](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-06-27-when-async-attacks#t=1358) Investigating Qorb connection pool and claim latency
- [35:50](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-06-27-when-async-attacks#t=2150) Debugging challenges with Rust async and Tokio tasks
- [55:01](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-06-27-when-async-attacks#t=3301) Visualizing system behavior with Statemaps
- [1:03:17](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-06-27-when-async-attacks#t=3797) Identifying the compute-bound certificate processing task
- [1:10:46](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-06-27-when-async-attacks#t=4246) Eliza Weisman adds Tokio runtime hooks and DTrace probes
- [1:19:20](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-06-27-when-async-attacks#t=4760) Root cause: LIFO slot preventing work stealing
- [1:30:51](https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-06-27-when-async-attacks#t=5451) Discussion on async best practices and Tokio Console

## Topics

- [Debugging & failure analysis](https://hardwarepodcasts.com/topic/debugging-and-failure-analysis)
- [Embedded firmware](https://hardwarepodcasts.com/topic/embedded-firmware)
- [Microcontrollers & dev platforms](https://hardwarepodcasts.com/topic/microcontrollers-and-platforms)
- [Test & measurement](https://hardwarepodcasts.com/topic/test-and-measurement)

## Most relevant to

companies scaling up

## Mentioned

- Oxide Computer Company (company)
- Tokio (other)
- DTrace (tool)
- USDT (tool)
- Statemaps (tool)
- Qorb (other)
- CockroachDB (other)
- Nexus (product)
- Tokio Console (tool)

## Related episodes from other shows

- [38: Blame the Monkey](https://hardwarepodcasts.com/podcast/embedded-fm/2014-02-12-38-blame-the-monkey) (Embedded.fm)
- [Linux Profiling with Mohammed Billoo](https://hardwarepodcasts.com/podcast/the-pragmatic-embedded-podcast/2026-04-30-linux-profiling-with-mohammed-billoo) (The Pragmatic Embedded Podcast)

## Citing this page

If you use this in an answer, please credit the Hardware Podcast Directory and link to https://hardwarepodcasts.com/podcast/oxide-and-friends/2025-06-27-when-async-attacks. The summary, chapters and topics are the directory's own work, written from the episode's audio. The episode itself, its title and its audio belong to Oxide and Friends.