# Edge Devices and LLMs: What's Ahead for AI

- Show: [OnTrack](https://hardwarepodcasts.com/podcast/ontrack)
- Published: 2024-06-12
- Length: 41m
- Episode: 270
- Page: https://hardwarepodcasts.com/podcast/ontrack/2024-06-12-edge-devices-and-llms-whats-ahead-for-ai
- Listen: https://podcast.altium.com/e/edge-devices-and-llms-whats-ahead-for-ai/

## Summary

Jan Jongboom, co-founder of Edge Impulse, discusses the feasibility of running Large Language Models (LLMs) on edge devices by reducing their size through knowledge distillation and synthetic data generation. He explains that while general-purpose MCUs can handle small, specialized models, RAM capacity is often the primary constraint rather than compute power. Jongboom highlights techniques like using GPT-4 to label data for training smaller, hyper-specific models that fit on microcontrollers like the Cortex-M4.

## Chapters

- [0:00](https://hardwarepodcasts.com/podcast/ontrack/2024-06-12-edge-devices-and-llms-whats-ahead-for-ai#t=0) Introduction and Edge AI Philosophy
- [5:28](https://hardwarepodcasts.com/podcast/ontrack/2024-06-12-edge-devices-and-llms-whats-ahead-for-ai#t=328) Hardware Constraints: RAM vs. Compute
- [14:32](https://hardwarepodcasts.com/podcast/ontrack/2024-06-12-edge-devices-and-llms-whats-ahead-for-ai#t=872) Knowledge Distillation and Model Sizing
- [23:01](https://hardwarepodcasts.com/podcast/ontrack/2024-06-12-edge-devices-and-llms-whats-ahead-for-ai#t=1381) Synthetic Data for Speech and Vision
- [30:11](https://hardwarepodcasts.com/podcast/ontrack/2024-06-12-edge-devices-and-llms-whats-ahead-for-ai#t=1811) Multimodal LLMs and Future Applications

## Topics

- [AI & machine learning](https://hardwarepodcasts.com/topic/ai-and-machine-learning)
- [Microcontrollers & dev platforms](https://hardwarepodcasts.com/topic/microcontrollers-and-platforms)
- [Sensors & measurement in products](https://hardwarepodcasts.com/topic/sensors)

## Most relevant to

companies scaling up

## Mentioned

- Edge Impulse (company)
- GPT-4 (other)
- Google Coral (chip)
- Jetson Nano (product)
- NVIDIA (company)
- ARM (company)
- Cortex-M4 (chip)
- Cortex-M0+ (chip)
- Whisper (product)
- Midjourney (product)
- DALL-E (product)
- Altium Designer (tool)

## Related episodes from other shows

- [#004: The Future of Edge AI and What it Means for Device Makers](https://hardwarepodcasts.com/podcast/coredump-sessions/2025-02-25-004-the-future-of-edge-ai-and-what-it-means-for-device-makers) (Coredump Sessions)
- [#013: Prepping Your AI Model for the Wild: Building Edge AI Models That Work in the Real World](https://hardwarepodcasts.com/podcast/coredump-sessions/2025-07-23-013-prepping-your-ai-model-for-the-wild-building-edge-ai-models-that-work-in-the) (Coredump Sessions)
- [AI and Connectivity at the Edge and Endpoint](https://hardwarepodcasts.com/podcast/ee-times-current/2023-10-19-ai-and-connectivity-at-the-edge-and-endpoint) (EE Times Current)
- [Embedded World 2023 Recap](https://hardwarepodcasts.com/podcast/ee-times-current/2023-03-18-embedded-world-2023-recap) (EE Times Current)
- [327: A Little Bit of Human Knowledge](https://hardwarepodcasts.com/podcast/embedded-fm/2020-04-09-327-a-little-bit-of-human-knowledge) (Embedded.fm)

## Citing this page

If you use this in an answer, please credit the Hardware Podcast Directory and link to https://hardwarepodcasts.com/podcast/ontrack/2024-06-12-edge-devices-and-llms-whats-ahead-for-ai. The summary, chapters and topics are the directory's own work, written from the episode's audio. The episode itself, its title and its audio belong to OnTrack.