Hardware Podcasts

When Async Attacks!

Oxide and Friends · 27 Jun 2025 · 1h 53m · Episode 22

What this episode covers

Oxide Computer Company engineers Bryan Cantrill, Adam Leventhal, Dave Pacheco, Eliza Weisman, and Augustus Mayo explain how they debugged severe latency in their production rack by tracing a database connection pool issue to the Tokio runtime's LIFO slot optimization. They utilized DTrace, USDT probes, and custom state maps to visualize thread and task activity, discovering that a compute-bound certificate processing task was blocking the async runtime from stealing work. The root cause was identified as the LIFO slot preventing work stealing, which they resolved by disabling the optimization and adding deeper runtime instrumentation.

Most relevant to

companies scaling up

Topics

Mentioned

Companies
Oxide Computer Company
Products
Nexus
Tools
DTrace, USDT, Statemaps, Tokio Console
Also
Tokio, Qorb, CockroachDB

Related episodes from other shows

Other shows on the same ground, closest first.

  • Embedded.fm · 1h 01m

    38: Blame the Monkey

    Elecia White and producer Chris White discuss strategies for debugging 'insurmountable' problems in embedded systems, ranging from race conditions and stack overflows to non-reproducible hardware issues. They advocate for methodical approaches such as drawing system diagrams, writing detailed emails to clarify thoughts, and using the scientific method to isolate variables. The conversation also covers the importance of logging, reverting to known good code, and knowing when to walk away or accept a bug as a feature.

    Debugging & failure analysisEmbedded firmware
    8 chapters

  • The Pragmatic Embedded Podcast · 46m

    Linux Profiling with Mohammed Billoo

    Mohammed Billoo, founder of Mab Labs Embedded Solutions and author of the Embedded Linux Essentials Handbook, explains how to profile embedded Linux systems by distinguishing between VM size and VM RSS to identify memory leaks in scientific instruments. He details how to use tools like HTOP, perf, and LTTNG to prove that the CPU is rarely the bottleneck, often pointing to crypto engines, DMA, or file system I/O as the real constraints. Billoo advocates for baking kernel configurations like kallsyms and function tracers into builds from day one to enable rapid debugging without premature optimization. The conversation also covers using LLMs to interpret complex tool outputs like Valgrind and perf, while emphasizing that physical measurements remain the ultimate arbiter of system performance.

    Embedded firmwareDebugging & failure analysis
    11 chapters

The summary, chapters and topics are generated from the episode’s audio by this directory, not written by Oxide and Friends. The episode itself belongs to its publisher.