Hardware Podcasts

Episode 19: The On-Device AI Debate

The Circuit · 22 May 2023 · 34m · Episode 20

What this episode covers

Ben Bajarin and Jay Goldberg discuss the economic and technical drivers shifting AI workloads from the cloud to on-device processing, highlighting companies like Qualcomm, Apple, and NVIDIA. They analyze the trade-offs between cloud-based model size and accuracy versus edge-based privacy and cost, noting that models like Llama and Stable Diffusion are being optimized for devices like the iPhone and Raspberry Pi. The hosts argue that while cloud AI offers superior capability, the high cost of inference and privacy concerns are pushing developers to adopt hybrid approaches and optimize silicon for power efficiency rather than raw throughput.

Most relevant to

large enterprises

Topics

Mentioned

Companies
Qualcomm, Apple, NVIDIA, Open AI, Google
Products
ChatGPT, Llama, Stable Diffusion, Raspberry Pi, Notion, Jasper, GPT, Bard

Related episodes from other shows

Other shows on the same ground, closest first.

  • The Next Byte Podcast · 46m

    235. Arm’s Vince Jesaitis on How AI Is Moving from the Cloud to Your Pocket

    Vince Jesaitis, Head of Global Government Affairs at Arm, explains how AI is shifting from cloud-based inference to local edge processing on devices like smartphones, earbuds, and cars. He details Arm's strategy of providing efficient CPU building blocks that enable high performance-per-watt, addressing energy and water constraints in data centers while enhancing user privacy and security. Jesaitis also discusses the critical role of government relations, including workforce development initiatives, the US Chips Act, and navigating divergent global regulations like the EU AI Act and GDPR.

    Connected products & IoTSecurity & privacy
    8 chapters

  • The Next Byte Podcast · 28m

    145. The Prius of Artificial Intelligence: Hybrid AI Models

    Hosts Daniel and Forbode discuss Qualcomm's proposal for 'hybrid AI,' a model that splits generative AI workloads between edge devices and the cloud to reduce cost, latency, and energy use. They explain how techniques like quantization and pruning allow large models to run locally on hardware like Snapdragon chips, citing examples in autonomous vehicles and mixed reality. The hosts also analyze the implications for hardware design, noting that rapid AI iteration may force faster hardware refresh cycles for consumer devices like iPhones.

    AI & machine learningMicrocontrollers & dev platforms
    7 chapters

  • EE Times Current · 25m

    Chips at the Edge: Innovation for AI at Edge and Automotive

    Hezzy Tzar, Executive Director at Synopsys, discusses the shift of AI workloads from the cloud to the edge, highlighting the critical balance of power, performance, and cost in devices like smartphones and automotive systems. He explains that memory bandwidth, specifically involving LPDDR6 and UFS, is the primary bottleneck for on-device AI, necessitating flexible multi-die architectures and standards like UCIe to manage scalability and reliability in automotive applications. Tzar emphasizes the importance of 'shift left' methodologies, using emulation and FPGA prototyping to handle rapid model changes, and predicts a future where hardware must support continuous software upgrades and subscription-based models for physical AI devices like robotics.

    AI & machine learningMicrocontrollers & dev platforms
    7 chapters

  • Over the Air · 24m

    AI at the Edge: Justin Schneck on the Future of Embedded Devices

    Justin Schneck, co-founder and CPO of Peridio, discusses the evolution of AI from cloud to edge, introducing Peridio's upcoming 'Avocado' OS designed to simplify embedded product development. He and host Ryan Prosser of Very explore the spectrum from cloud AI to embedded AI, noting how specialized silicon and miniaturized models are enabling practical applications like Garmin's radar-enabled bike lights. Schneck argues that while cloud AI handles massive compute, the real value lies in robust, self-initializing embedded systems that reduce hardware complexity for developers.

    Embedded firmwareAI & machine learning
    7 chapters

  • IoT For All Podcast · 21m

    The State of AIoT | Transforma Insights' Jim Morrish | Internet of Things Podcast

    Jim Morrish, Founding Partner at Transforma Insights, defines AIoT as the specific deployment of AI algorithms on IoT devices rather than in the cloud, enabling benefits like reduced latency, privacy, and lower bandwidth usage. He highlights key use cases such as smart speakers, CCTV for defect detection, and robotics, noting that while the cloud AI market (exemplified by Nvidia) is larger, the AIoT hardware market is fragmented with specialized chipmakers like Semid, Mythic, and Halo. Morrish argues that success requires optimizing solutions for specific use cases, managing device drift in constrained environments, and navigating emerging regulatory frameworks around fairness and liability.

    Connected products & IoTMicrocontrollers & dev platforms
    8 chapters

The summary, chapters and topics are generated from the episode’s audio by this directory, not written by The Circuit. The episode itself belongs to its publisher.