Hardware Podcasts

#464: Why LLM Unpredictability is a Liability in MedTech

Global Medical Device Podcast · 13 Jul 2026 · 53m · Episode 464

What this episode covers

Tyler Harmon, CEO of Iaso Automated Medical Systems, explains why the inherent randomness in Large Language Models (LLMs) makes them unsuitable for safety-critical medical devices compared to deterministic algorithms like Convolutional Neural Networks (CNNs). He details how Iaso uses non-LLM machine learning to detect Acute Respiratory Distress Syndrome (ARDS) in ICU patients and advocates for treating third-party AI stacks as Software of Unknown Provenance (SOUP). Harmon also outlines the need for a dedicated Machine Learning Management System (AIMS) within Quality Management Systems to govern AI governance, hallucination risks, and data integrity.

Most relevant to

early-stage companies

Topics

Mentioned

Companies
Iaso Automated Medical Systems, Google, AWS, Apple, Drager, Fisher, Johnson & Johnson, Georgia Tech, Blue Goat, HIS AI
Products
ChatGPT, Claude, Amazon Alexa, Site AI, Gemini
Also
PubMed, Medline, Google Scholar, IEC 62304, ISO 27001

Related episodes from other shows

Other shows on the same ground, closest first.

  • Embedded.fm · 1h 16m

    421: Paint the Iceberg Yellow

    Chris Hobbs, who works in kernel development at BlackBerry QNX and authored 'Embedded Software Development for Safety-Critical Systems,' explains the distinction between safety-critical and standard embedded systems, emphasizing safety culture and the concept of SOTIF (Safety of the Intended Functionality). He details how modern systems, such as autonomous vehicles and medical devices, face risks from 'accidental systems' where interactions between components create unforeseen hazards, citing examples like a radar system dependent on jammed GPS. Hobbs advocates for goal-based standards like UL 4600 and the use of Safety Cases with Goal Structuring Notation to demonstrate safety rather than relying on prescriptive checklists. He also discusses the challenges of verifying machine learning algorithms, the limitations of languages like C and Rust in safety contexts, and the importance of structured risk analysis using tools like Bayesian fault trees and LaTeX for traceability.

    Embedded firmwareSecurity & privacy
    12 chapters

  • The Pragmatic Embedded Podcast · 55m

    Crossover with Embedded AI Podcast

    Jeff Gable, Luca, and Ryan Torvik discuss integrating Large Language Models (LLMs) into embedded systems engineering workflows, emphasizing that AI must augment rather than replace rigorous engineering discipline. They identify 'vibe coding' as a critical anti-pattern where developers lose control of architectural decisions by letting AI generate too much code at once, leading to unmanageable outputs and hallucinations. The hosts advocate for strict context management, test-driven development (TDD), and small, incremental commits to maintain deterministic quality, particularly in safety-critical domains like medical devices and automotive systems. They conclude that while LLMs accelerate prototyping and boilerplate generation, human code review and ownership remain non-negotiable for reliable production software.

    Embedded firmwareDebugging & failure analysis
    8 chapters

  • EE Times Current · 29m

    Building a Framework to Trust AI

    Helen Toner, Director of Strategy at Georgetown University’s Center for Security and Emerging Technology, argues that AI currently lacks the engineering discipline required for mission-critical systems, describing current machine learning development as akin to 'alchemy.' She outlines a necessary framework involving basic science, socio-legal standards, and engineering rigor to ensure reliability, citing failures in social media algorithms and hiring tools as evidence of current risks. The episode also covers industry news, including Arm’s Sophie architecture for automotive software and Silicon Labs’ new in-line programming capabilities for IoT security.

    AI & machine learningSecurity & privacy
    5 chapters

  • Embedded.fm · 1h 08m

    502: Chat, J'ai Peté!

    Elecia White and Christopher White discuss the TV adaptation of Martha Wells' 'Murderbot Diaries' and the deceptive nature of Large Language Models (LLMs) like ChatGPT, Claude, and Gemini. They highlight specific failures of AI, including lawyers citing non-existent legal precedents and GitHub Copilot agents generating endless, looping pull requests. The conversation also covers control systems, contrasting theoretical 'fancy math' like Kalman filters with empirical tuning, and mentions the Segway patent US7370713B1.

    AI & machine learningEmbedded firmware
    6 chapters

  • Oxide and Friends · 1h 23m

    Adversarial Machine Learning

    Nicholas Carlini, a security researcher specializing in adversarial machine learning, joins Bryan Cantrill and Adam Leventhal to discuss how sequences of seemingly random tokens can cause Large Language Models (LLMs) to ignore safety restrictions. Carlini explains that these attacks, which include 'poisoning' training data by exploiting expired domain names and using adversarial suffixes to bypass jailbreaks, demonstrate that LLMs are vulnerable to inputs that look like noise but trigger specific internal features. The conversation highlights the transferability of these attacks across different models, the importance of open-source models for security research, and the practical risks of deploying LLMs without understanding their failure modes.

    Security & privacyAI & machine learning
    8 chapters

The summary, chapters and topics are generated from the episode’s audio by this directory, not written by Global Medical Device Podcast. The episode itself belongs to its publisher.