The Pragmatic Embedded Podcast · 30 Apr 2026 · 46m
Mohammed Billoo, founder of Mab Labs Embedded Solutions and author of the Embedded Linux Essentials Handbook, explains how to profile embedded Linux systems by distinguishing between VM size and VM RSS to identify memory leaks in scientific instruments. He details how to use tools like HTOP, perf, and LTTNG to prove that the CPU is rarely the bottleneck, often pointing to crypto engines, DMA, or file system I/O as the real constraints. Billoo advocates for baking kernel configurations like kallsyms and function tracers into builds from day one to enable rapid debugging without premature optimization. The conversation also covers using LLMs to interpret complex tool outputs like Valgrind and perf, while emphasizing that physical measurements remain the ultimate arbiter of system performance.
Where this conversation describes AI being used to build or deliver a product. Part of the AI in practice tracker.
Engineering & simulation
The speaker uses LLMs to interpret complex tool outputs (like Valgrind) and identify false positives or recommend specific hooks/options to enable for accurate memory leak detection.
“that's where like an LLM could come in handy … You give it, you know, snippets of your source code. … and it can tell you like, well, you know, this looks like this is a false positive.”
Speaker 2 · Hear it at 38:32
early-stage companies
Other shows on the same ground, closest first.
Julia Evans, a software engineer at Stripe, discusses using Linux system-level tools like `strace`, `perf`, and `top` to debug applications as black boxes. She explains how `strace` reveals system calls to understand program behavior without reading source code, and how `perf top` identifies CPU-intensive functions like garbage collection. Evans also shares her philosophy on debugging, emphasizing the importance of maintaining confidence and understanding that computers are logical machines, even when bugs seem impossible.
Jacob Beningo, an embedded software consultant and author, discusses the critical role of tracing technology in embedded development, recommending tools like Percepio Tracealyzer and SEGGER SystemView to visualize system behavior without the performance overhead of print statements. He explains how tracing helps debug timing issues, optimize low-power states, and validate real-time operating system (RTOS) performance, particularly in complex applications like satellite flight software. Beningo also shares insights from his books on reusable firmware and MicroPython, advocating for the use of Hardware Abstraction Layers (HALs) to accelerate development while understanding their limitations, and highlights the Embedded Online Conference as a way to access industry expertise efficiently.
Elecia White and producer Chris White discuss strategies for debugging 'insurmountable' problems in embedded systems, ranging from race conditions and stack overflows to non-reproducible hardware issues. They advocate for methodical approaches such as drawing system diagrams, writing detailed emails to clarify thoughts, and using the scientific method to isolate variables. The conversation also covers the importance of logging, reverting to known good code, and knowing when to walk away or accept a bug as a feature.
Cliff Biffle, principal engineer at Oxide Computer Company, explains the origins of Hubris, an embedded operating system designed for Oxide's rack-scale computers, drawing on his prior experience with Project Loon at Google. The team evaluated existing options like Tock and QNX but chose to build Hubris in Rust to ensure memory safety and value alignment, leveraging a software-hardware co-design approach to bypass the risks of secure FPGAs. Key technical decisions include using a minimal kernel, self-describing build archives, and advanced debugging tools like Humility to manage complex embedded systems without dynamic loading or self-hosting constraints.
Hosts Elecia White and Christopher White discuss the hidden complexities of embedded software engineering in preparation for Elecia's upcoming IEEE presentation on 'The Tricky Parts' of embedded systems. They cover specific technical challenges including the difficulty of debugging with limited visibility, the counterintuitive nature of power optimization (such as balancing sleep states against wake-up costs), and the pain of managing unstable SDKs and sample code from vendors like Nordic and NXP. The conversation also touches on the necessity of understanding map files for memory management and the importance of using test equipment like oscilloscopes for accurate timing analysis.
The summary, chapters and topics are generated from the episode’s audio by this directory, not written by The Pragmatic Embedded Podcast. The episode itself belongs to its publisher.