Hardware Podcasts

145. The Prius of Artificial Intelligence: Hybrid AI Models

The Next Byte Podcast · 24 Oct 2023 · 28m

What this episode covers

Hosts Daniel and Forbode discuss Qualcomm's proposal for 'hybrid AI,' a model that splits generative AI workloads between edge devices and the cloud to reduce cost, latency, and energy use. They explain how techniques like quantization and pruning allow large models to run locally on hardware like Snapdragon chips, citing examples in autonomous vehicles and mixed reality. The hosts also analyze the implications for hardware design, noting that rapid AI iteration may force faster hardware refresh cycles for consumer devices like iPhones.

Most relevant to

large enterprises

Topics

Mentioned

Companies
Qualcomm, OpenAI, Microsoft Azure, NVIDIA, Honda, Volvo
Products
ChatGPT, Stable Diffusion
Chips
Snapdragon
Tools
TensorFlow Lite

Related episodes from other shows

Other shows on the same ground, closest first.

  • The Circuit · 34m

    Episode 19: The On-Device AI Debate

    Ben Bajarin and Jay Goldberg discuss the economic and technical drivers shifting AI workloads from the cloud to on-device processing, highlighting companies like Qualcomm, Apple, and NVIDIA. They analyze the trade-offs between cloud-based model size and accuracy versus edge-based privacy and cost, noting that models like Llama and Stable Diffusion are being optimized for devices like the iPhone and Raspberry Pi. The hosts argue that while cloud AI offers superior capability, the high cost of inference and privacy concerns are pushing developers to adopt hybrid approaches and optimize silicon for power efficiency rather than raw throughput.

    Connected products & IoTMicrocontrollers & dev platforms
    10 chapters

  • EE Times Current · 29m

    AI and Connectivity at the Edge and Endpoint

    Kaushal Vora and Mo Dogar from Renesas discuss the convergence of AI and IoT (AIoT) at the edge, distinguishing between resource-intensive generative AI and predictive AI suited for constrained endpoint devices. They highlight Renesas' acquisition of Reality AI and its AutoML toolchain, which allows embedded engineers to optimize machine learning models for microcontrollers without adding hardware sensors, as demonstrated in washer and battery management use cases. The guests explain how decentralizing intelligence reduces latency, power consumption, and security risks by performing inference locally on devices like the RH850 microcontroller.

    Embedded firmwareMicrocontrollers & dev platforms
    8 chapters

  • EE Times Current · 25m

    Chips at the Edge: Innovation for AI at Edge and Automotive

    Hezzy Tzar, Executive Director at Synopsys, discusses the shift of AI workloads from the cloud to the edge, highlighting the critical balance of power, performance, and cost in devices like smartphones and automotive systems. He explains that memory bandwidth, specifically involving LPDDR6 and UFS, is the primary bottleneck for on-device AI, necessitating flexible multi-die architectures and standards like UCIe to manage scalability and reliability in automotive applications. Tzar emphasizes the importance of 'shift left' methodologies, using emulation and FPGA prototyping to handle rapid model changes, and predicts a future where hardware must support continuous software upgrades and subscription-based models for physical AI devices like robotics.

    AI & machine learningMicrocontrollers & dev platforms
    7 chapters

  • Catalyst · 47m

    Will inference move to the edge?

    Dr. Ben Lee, a professor at the University of Pennsylvania and visiting researcher at Google, discusses the potential shift of AI inference compute from centralized hyperscale data centers to the edge and on-device. He explains that while training remains centralized due to massive GPU coordination needs, inference is amenable to distributed locations because it lacks the frequent inter-GPU communication spikes that cause power grid instability. Lee predicts that by 2035, up to 80% of inference could occur locally, primarily at edge data centers (10-50 MW) rather than on consumer devices, due to the trade-off between latency/privacy and the hardware constraints of shrinking model capabilities for mobile silicon.

    Energy & climate techAI & machine learning
    6 chapters

  • EE Times Current · 45m

    From Data Centers to Devices – The Ongoing [R]Evolution of AI Computing

    Matt Gutierrez, Senior Director of Marketing at Synopsys, discusses the migration of AI computing from centralized data centers to the network edge. He explains that this shift is driven by the need for low latency, data privacy, and power efficiency, enabled by smaller neural networks and specialized hardware like NPUs and TPUs. Gutierrez details how Synopsys supports this through its ARC processor IP portfolio and EDA tools, helping engineers manage the complexity of heterogeneous processing platforms for edge devices.

    AI & machine learningMicrocontrollers & dev platforms
    7 chapters

The summary, chapters and topics are generated from the episode’s audio by this directory, not written by The Next Byte Podcast. The episode itself belongs to its publisher.