Hardware Podcasts

Will inference move to the edge?

Catalyst · 18 Dec 2025 · 47m

What this episode covers

Dr. Ben Lee, a professor at the University of Pennsylvania and visiting researcher at Google, discusses the potential shift of AI inference compute from centralized hyperscale data centers to the edge and on-device. He explains that while training remains centralized due to massive GPU coordination needs, inference is amenable to distributed locations because it lacks the frequent inter-GPU communication spikes that cause power grid instability. Lee predicts that by 2035, up to 80% of inference could occur locally, primarily at edge data centers (10-50 MW) rather than on consumer devices, due to the trade-off between latency/privacy and the hardware constraints of shrinking model capabilities for mobile silicon.

Most relevant to

large enterprises

Topics

Mentioned

Companies
Microsoft, Google, Amazon, Meta, Apple
Products
GPT, Claude, Dropbox

Related episodes from other shows

Other shows on the same ground, closest first.

  • EE Times Current · 45m

    From Data Centers to Devices – The Ongoing [R]Evolution of AI Computing

    Matt Gutierrez, Senior Director of Marketing at Synopsys, discusses the migration of AI computing from centralized data centers to the network edge. He explains that this shift is driven by the need for low latency, data privacy, and power efficiency, enabled by smaller neural networks and specialized hardware like NPUs and TPUs. Gutierrez details how Synopsys supports this through its ARC processor IP portfolio and EDA tools, helping engineers manage the complexity of heterogeneous processing platforms for edge devices.

    AI & machine learningMicrocontrollers & dev platforms
    7 chapters

  • The Next Byte Podcast · 46m

    235. Arm’s Vince Jesaitis on How AI Is Moving from the Cloud to Your Pocket

    Vince Jesaitis, Head of Global Government Affairs at Arm, explains how AI is shifting from cloud-based inference to local edge processing on devices like smartphones, earbuds, and cars. He details Arm's strategy of providing efficient CPU building blocks that enable high performance-per-watt, addressing energy and water constraints in data centers while enhancing user privacy and security. Jesaitis also discusses the critical role of government relations, including workforce development initiatives, the US Chips Act, and navigating divergent global regulations like the EU AI Act and GDPR.

    Connected products & IoTSecurity & privacy
    8 chapters

  • The Circuit · 34m

    Episode 19: The On-Device AI Debate

    Ben Bajarin and Jay Goldberg discuss the economic and technical drivers shifting AI workloads from the cloud to on-device processing, highlighting companies like Qualcomm, Apple, and NVIDIA. They analyze the trade-offs between cloud-based model size and accuracy versus edge-based privacy and cost, noting that models like Llama and Stable Diffusion are being optimized for devices like the iPhone and Raspberry Pi. The hosts argue that while cloud AI offers superior capability, the high cost of inference and privacy concerns are pushing developers to adopt hybrid approaches and optimize silicon for power efficiency rather than raw throughput.

    Connected products & IoTMicrocontrollers & dev platforms
    10 chapters

  • The Circuit · 38m

    Episode 8: AI Infrastructure and Silicon Demands

    Ben Bajarin and Jay Goldberg discuss the immense compute demands of training large foundational models like GPT-3 and GPT-4, noting that Nvidia’s A100 and H100 GPUs dominate this $300 billion infrastructure market due to CUDA’s software moat. They analyze the bottleneck in inference scalability, arguing that while training is a small fraction of the market, inference costs will drop as incumbents like Google, Amazon, and Apple leverage their own silicon, leaving little room for new AI accelerator startups. The hosts conclude that generative AI represents a powerful new category of productivity software rather than a fundamental shift in human-computer interaction, driving data center Capex through natural upgrade cycles rather than complete rebuilds.

    Chips, FPGAs & semiconductorsConnected products & IoT
    8 chapters

  • The Circuit · 43m

    Episode 120: NVIDIA Earnings, The Future of AI inference, China AI and more

    Austin Lyons joins Jay Goldberg to analyze NVIDIA's recent earnings, highlighting the shift from training to inference workloads and the impact of US export bans on China revenue. They discuss the competitive landscape for AI inference, including on-device processing, enterprise AI factories, and the strategic moves by competitors like Google and Dell. The conversation explores the challenges of realizing ROI from massive AI infrastructure investments and the evolving consumer and enterprise use cases for generative AI.

    Connected products & IoTPricing, margins & unit economics
    7 chapters

The summary, chapters and topics are generated from the episode’s audio by this directory, not written by Catalyst. The episode itself belongs to its publisher.