395: I Can No Longer Play Ping Pong

Embedded.fm · 10 Dec 2021 · 1h 17m · Episode 395

What this episode covers

Tyler Hoffman, co-founder of Memfault, explains how his team at Pebble and later Fitbit transformed firmware development by standardizing developer environments, optimizing build systems, and automating debugging. He details the shift from slow, sequential IAR builds to parallel GCC builds using Make and Conda, which reduced build times from 18 minutes to 2 minutes. Hoffman also describes building web applications and GDB Python scripts to enable remote debugging of field crashes without physical debuggers. The discussion highlights the importance of dedicated developer productivity roles and the evolution of tools from one-off scripts to robust, company-wide platforms.

Mentioned

Companies
Memfault, Pebble, Fitbit
Tools
IAR, GCC, Conda, Invoke, fzf, GDB, Eclipse, VS Code, CppUTest, ESP-IDF, Zephyr, Nordic SDK

Other shows on the same ground, closest first.

  • Coredump Sessions · 1h 13m

    #006: Pebble’s Code is Free: Three Former Pebble Engineers Discuss Why It's Important (PART 2/2)

    Brad Murray, former head of Pebble firmware, and Tomas, former developer evangelist at Pebble, discuss the risks and decisions behind building firmware for Pebble smartwatches. They detail a critical OTA update bug that nearly bricked nearly a million devices, which was recovered using a clever BLE-based stack overflow hack. The team also explains how they built a massive developer ecosystem through tools like CloudPebble and the Watch Face Generator, and how they optimized code size to backport Pebble 3.0 to legacy hardware using techniques like Link Time Optimization and bootloader rewriting.

  • The Pragmatic Embedded Podcast · 53m

    François Baldassari from Memfault on IoT Observability

    François Baldassari, founder and CEO of Memfault, discusses the evolution of embedded software and the critical need for IoT observability tools. He recounts his experience at Pebble, where a lack of field data led to a severe firmware update bug affecting Android users, which inspired the creation of Memfault's crash reporting and OTA solutions. Baldassari argues that modern embedded teams must adopt agile practices, such as decoupling software from hardware via emulators and implementing 'day 0 updates,' to iterate effectively. He highlights persistent industry challenges, including C language memory management issues, code space constraints, and the emerging crisis of embedded security and privacy.

  • The Amp Hour · 1h 20m

    Firmware Deployment and Troubleshooting with Akbar Dhanaliwala

    Chris Gammell, founder of Lager Data, explains how his experiences with companies like SolarCity, Ringly, and Minna Life revealed the tedious nature of embedded firmware testing, inspiring the creation of the Lager Gateway platform. He details how the system enables continuous integration and deployment (CI/CD) for embedded devices by allowing engineers to run unit and system tests, flash firmware, and debug remotely from a command line without physical lab access. Gammell highlights the use of Dockerized build environments and integration with GitHub Actions to automate testing, ensuring that changes to firmware do not break device functionality or over-the-air update capabilities.

  • Coredump Sessions · 49m

    #003: Pebble's Code is Free: Three Former Pebble Engineers Discuss Why It's Important (PART 1/2)

    Francois Baldassari, Chris Coleman, and Brad Murray, former engineers at Pebble, discuss the technical architecture and engineering decisions behind PebbleOS. They explain how a single codebase supported multiple hardware variants, enabling a platform approach that simplified updates and app compatibility. The team details their custom NOR Flash file system, app sandboxing using MPU fault handlers, and the strategic use of intentional crashes to generate core dumps for debugging. They also highlight the importance of analytics, serial protocols, and the collaboration between firmware and hardware teams in shaping the final product.

  • The Pragmatic Embedded Podcast · 58m

    Introducing Agile at John Deere with Joe Schneider

    Joe Schneider, founder of Dojo 5, recounts his experience modernizing firmware development at John Deere, where he led a grassroots technical transformation from a legacy Star Team environment to Git, Cruise Control, and eventually Agile methodologies. He details how overcoming specific engineering pain points, such as 45-minute build times and inefficient GPS simulators, built the credibility needed to advocate for organizational changes like cross-functional teams and external coaching. Schneider now runs Dojo 5, which helps other embedded companies bridge the gap between web/mobile practices and firmware development through their EmbedOps platform and services.

The summary, chapters and topics are generated from the episode’s audio by this directory, not written by Embedded.fm. The episode itself belongs to its publisher.