Hardware Podcasts

Edge Devices and LLMs: What's Ahead for AI

OnTrack · 12 Jun 2024 · 41m · Episode 270

What this episode covers

Jan Jongboom, co-founder of Edge Impulse, discusses the feasibility of running Large Language Models (LLMs) on edge devices by reducing their size through knowledge distillation and synthetic data generation. He explains that while general-purpose MCUs can handle small, specialized models, RAM capacity is often the primary constraint rather than compute power. Jongboom highlights techniques like using GPT-4 to label data for training smaller, hyper-specific models that fit on microcontrollers like the Cortex-M4.

Most relevant to

companies scaling up

Topics

Mentioned

Companies
Edge Impulse, NVIDIA, ARM
Products
Jetson Nano, Whisper, Midjourney, DALL-E
Chips
Google Coral, Cortex-M4, Cortex-M0+
Tools
Altium Designer
Also
GPT-4

Related episodes from other shows

Other shows on the same ground, closest first.

  • Coredump Sessions · 58m

    #004: The Future of Edge AI and What it Means for Device Makers

    Alex Samuelson, CTO and co-founder of Imagimob, joins Memfault founders Francois Baldassari and Chris Coleman to discuss the practical implementation of Edge AI on embedded devices. They highlight how hardware accelerators like the Cortex-M55 and U55 have enabled GPU-like performance on microcontrollers, allowing for real-time inference in applications such as welding quality monitoring and fall detection. The conversation emphasizes that data collection, labeling, and memory constraints are the primary bottlenecks, requiring device makers to build data acquisition firmware early in the development process. Finally, they discuss the shift toward 'AI as a Service' business models, where hardware companies generate recurring revenue by updating models over the device's lifecycle.

    AI & machine learningMicrocontrollers & dev platforms
    8 chapters

  • Coredump Sessions · 57m

    #013: Prepping Your AI Model for the Wild: Building Edge AI Models That Work in the Real World

    David Tischler and Alessandro Grande from Edge Impulse discuss the practical realities of deploying Edge AI models on constrained devices, emphasizing that customization and fine-tuning are the primary value propositions over raw model complexity. They explain how effective Edge AI requires a combination of traditional DSP pre-processing and ML models, and why hardware selection must balance compute power with software ecosystem usability. The conversation highlights the critical need for observability and OTA updates to monitor model drift and push fixes in the field, as models rarely work perfectly out of the box.

    AI & machine learningMicrocontrollers & dev platforms
    7 chapters

  • EE Times Current · 29m

    AI and Connectivity at the Edge and Endpoint

    Kaushal Vora and Mo Dogar from Renesas discuss the convergence of AI and IoT (AIoT) at the edge, distinguishing between resource-intensive generative AI and predictive AI suited for constrained endpoint devices. They highlight Renesas' acquisition of Reality AI and its AutoML toolchain, which allows embedded engineers to optimize machine learning models for microcontrollers without adding hardware sensors, as demonstrated in washer and battery management use cases. The guests explain how decentralizing intelligence reduces latency, power consumption, and security risks by performing inference locally on devices like the RH850 microcontroller.

    Embedded firmwareMicrocontrollers & dev platforms
    8 chapters

  • EE Times Current · 22m

    Embedded World 2023 Recap

    Nitin Dahad, Francoise Pelle, and Sally Ward Foxton of EE Times recap Embedded World 2023, highlighting the shift toward edge AI, tiny ML, and software-defined vehicles. They discuss specific implementations from companies like Renaissance, STMicroelectronics, and Silicon Labs, focusing on running AI models on Cortex-M85 and STM32 microcontrollers with low power consumption. The episode also covers security initiatives from Kudosip and Trust In Software, as well as neuromorphic computing demos from InnerTense and Brain Chip.

    Embedded firmwareSecurity & privacy
    8 chapters

  • Embedded.fm · 1h 23m

    327: A Little Bit of Human Knowledge

    Daniel Situnayake, author of 'TinyML' and engineer at Edge Impulse, explains how to run machine learning models on ultra-low-power microcontrollers using TensorFlow Lite for Microcontrollers. He details the workflow of using Google Colab to train models and deploying them to boards like the Arduino Nano 33 BLE Sense and SparkFun Edge. Situnayake emphasizes that success depends on data quality and scoping, noting that while the technology is shippable, it requires careful handling of diverse datasets to avoid poor performance in real-world conditions.

    Embedded firmwareAI & machine learning
    8 chapters

The summary, chapters and topics are generated from the episode’s audio by this directory, not written by OnTrack. The episode itself belongs to its publisher.