Hardware Podcasts

Mr. Nagle's Wild Ride

Oxide and Friends · 3 Oct 2023 · 1h 30m · Episode 27

What this episode covers

Bryan Cantrill of Oxide Computer Company details his debugging journey into network latency outliers, using bpftrace and Rust to visualize IO patterns and identify the root cause as the interaction between Nagle's algorithm and delayed ACKs. He explains how the combination of these two TCP defaults created 100ms latency bands in their virtual storage service, a problem exacerbated by the lack of visibility in Linux debugging tools. The episode features insights from Tom Lyon, Dan Ports, Eliza Weisman, and Shubh La, who discuss the historical context of these algorithms, the importance of TCP_NODELAY, and the need for better system observability to catch such transient issues.

Most relevant to

early-stage companies

Topics

Mentioned

Companies
Oxide Computer Company
Products
Honeycomb
Tools
bpftrace, DTrace, Wireshark
Also
Rust, Tokio

Related episodes from other shows

Other shows on the same ground, closest first.

  • Embedded.fm · 58m

    141: Malevolent and Trying to Trick You

    Julia Evans, a software engineer at Stripe, discusses using Linux system-level tools like `strace`, `perf`, and `top` to debug applications as black boxes. She explains how `strace` reveals system calls to understand program behavior without reading source code, and how `perf top` identifies CPU-intensive functions like garbage collection. Evans also shares her philosophy on debugging, emphasizing the importance of maintaining confidence and understanding that computers are logical machines, even when bugs seem impossible.

    Debugging & failure analysisTest & measurement
    7 chapters

  • Embedded.fm · 1h 09m

    364: All the Abstractions

    Jacob Beningo, an embedded software consultant and author, discusses the critical role of tracing technology in embedded development, recommending tools like Percepio Tracealyzer and SEGGER SystemView to visualize system behavior without the performance overhead of print statements. He explains how tracing helps debug timing issues, optimize low-power states, and validate real-time operating system (RTOS) performance, particularly in complex applications like satellite flight software. Beningo also shares insights from his books on reusable firmware and MicroPython, advocating for the use of Hardware Abstraction Layers (HALs) to accelerate development while understanding their limitations, and highlights the Embedded Online Conference as a way to access industry expertise efficiently.

    Embedded firmwareMicrocontrollers & dev platforms
    9 chapters

  • The Pragmatic Embedded Podcast · 46m

    Linux Profiling with Mohammed Billoo

    Mohammed Billoo, founder of Mab Labs Embedded Solutions and author of the Embedded Linux Essentials Handbook, explains how to profile embedded Linux systems by distinguishing between VM size and VM RSS to identify memory leaks in scientific instruments. He details how to use tools like HTOP, perf, and LTTNG to prove that the CPU is rarely the bottleneck, often pointing to crypto engines, DMA, or file system I/O as the real constraints. Billoo advocates for baking kernel configurations like kallsyms and function tracers into builds from day one to enable rapid debugging without premature optimization. The conversation also covers using LLMs to interpret complex tool outputs like Valgrind and perf, while emphasizing that physical measurements remain the ultimate arbiter of system performance.

    Embedded firmwareDebugging & failure analysis
    11 chapters

The summary, chapters and topics are generated from the episode’s audio by this directory, not written by Oxide and Friends. The episode itself belongs to its publisher.