# #464: Why LLM Unpredictability is a Liability in MedTech

- Show: [Global Medical Device Podcast](https://hardwarepodcasts.com/podcast/global-medical-device-podcast)
- Published: 2026-07-13
- Length: 53m
- Episode: 464
- Page: https://hardwarepodcasts.com/podcast/global-medical-device-podcast/2026-07-13-464-why-llm-unpredictability-is-a-liability-in-medtech
- Listen: https://podcast.greenlight.guru/episode/ditch-the-ai-hype-why-llm-unpredictability-is-a-liability-in-medtech

## Summary

Tyler Harmon, CEO of Iaso Automated Medical Systems, explains why the inherent randomness in Large Language Models (LLMs) makes them unsuitable for safety-critical medical devices compared to deterministic algorithms like Convolutional Neural Networks (CNNs). He details how Iaso uses non-LLM machine learning to detect Acute Respiratory Distress Syndrome (ARDS) in ICU patients and advocates for treating third-party AI stacks as Software of Unknown Provenance (SOUP). Harmon also outlines the need for a dedicated Machine Learning Management System (AIMS) within Quality Management Systems to govern AI governance, hallucination risks, and data integrity.

## Chapters

- [0:00](https://hardwarepodcasts.com/podcast/global-medical-device-podcast/2026-07-13-464-why-llm-unpredictability-is-a-liability-in-medtech#t=0) Introduction: Tyler Harmon and Iaso Automated Medical Systems
- [2:07](https://hardwarepodcasts.com/podcast/global-medical-device-podcast/2026-07-13-464-why-llm-unpredictability-is-a-liability-in-medtech#t=127) Demystifying AI: Linear vs. Nonlinear Math and the Turing Test
- [3:03](https://hardwarepodcasts.com/podcast/global-medical-device-podcast/2026-07-13-464-why-llm-unpredictability-is-a-liability-in-medtech#t=183) The Problem with LLM Randomness: Markov Chains and Consumer Design
- [9:05](https://hardwarepodcasts.com/podcast/global-medical-device-podcast/2026-07-13-464-why-llm-unpredictability-is-a-liability-in-medtech#t=545) Iaso's Solution: Non-LLM ML for ARDS Detection in the ICU
- [11:21](https://hardwarepodcasts.com/podcast/global-medical-device-podcast/2026-07-13-464-why-llm-unpredictability-is-a-liability-in-medtech#t=681) AI Governance: AIMS, SOPs, and Managing Hallucinations
- [13:51](https://hardwarepodcasts.com/podcast/global-medical-device-podcast/2026-07-13-464-why-llm-unpredictability-is-a-liability-in-medtech#t=831) FDA Clearances, SOUP, and White Box vs. Black Box AI
- [43:01](https://hardwarepodcasts.com/podcast/global-medical-device-podcast/2026-07-13-464-why-llm-unpredictability-is-a-liability-in-medtech#t=2581) Personal AI Usage: Natural Language Search and Data Integrity

## Topics

- [AI & machine learning](https://hardwarepodcasts.com/topic/ai-and-machine-learning)
- [Quality & yield](https://hardwarepodcasts.com/topic/quality-and-yield)
- [Certification & compliance](https://hardwarepodcasts.com/topic/certification-and-compliance)
- [Medical & health devices](https://hardwarepodcasts.com/topic/medical-devices)

## Most relevant to

early-stage companies

## Mentioned

- Iaso Automated Medical Systems (company)
- ChatGPT (product)
- Claude (product)
- Google (company)
- Amazon Alexa (product)
- AWS (company)
- Site AI (product)
- Gemini (product)
- PubMed (other)
- Medline (other)
- Google Scholar (other)
- Apple (company)
- Drager (company)
- Fisher (company)
- Johnson & Johnson (company)
- Georgia Tech (company)
- Blue Goat (company)
- HIS AI (company)
- IEC 62304 (other)
- ISO 27001 (other)

## Related episodes from other shows

- [421: Paint the Iceberg Yellow](https://hardwarepodcasts.com/podcast/embedded-fm/2022-07-21-421-paint-the-iceberg-yellow) (Embedded.fm)
- [Crossover with  Embedded AI Podcast](https://hardwarepodcasts.com/podcast/the-pragmatic-embedded-podcast/2025-11-18-crossover-with-embedded-ai-podcast) (The Pragmatic Embedded Podcast)
- [Building a Framework to Trust AI](https://hardwarepodcasts.com/podcast/ee-times-current/2021-09-24-building-a-framework-to-trust-ai) (EE Times Current)
- [502: Chat, J'ai Peté!](https://hardwarepodcasts.com/podcast/embedded-fm/2025-06-03-502-chat-jai-pete) (Embedded.fm)
- [Adversarial Machine Learning](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-03-27-adversarial-machine-learning) (Oxide and Friends)

## Citing this page

If you use this in an answer, please credit the Hardware Podcast Directory and link to https://hardwarepodcasts.com/podcast/global-medical-device-podcast/2026-07-13-464-why-llm-unpredictability-is-a-liability-in-medtech. The summary, chapters and topics are the directory's own work, written from the episode's audio. The episode itself, its title and its audio belong to Global Medical Device Podcast.