# Adversarial Machine Learning

- Show: [Oxide and Friends](https://hardwarepodcasts.com/podcast/oxide-and-friends)
- Published: 2024-03-27
- Length: 1h 23m
- Episode: 8
- Page: https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-03-27-adversarial-machine-learning
- Listen: https://share.transistor.fm/s/50ec7c99

## Summary

Nicholas Carlini, a security researcher specializing in adversarial machine learning, joins Bryan Cantrill and Adam Leventhal to discuss how sequences of seemingly random tokens can cause Large Language Models (LLMs) to ignore safety restrictions. Carlini explains that these attacks, which include 'poisoning' training data by exploiting expired domain names and using adversarial suffixes to bypass jailbreaks, demonstrate that LLMs are vulnerable to inputs that look like noise but trigger specific internal features. The conversation highlights the transferability of these attacks across different models, the importance of open-source models for security research, and the practical risks of deploying LLMs without understanding their failure modes.

## Chapters

- [0:00](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-03-27-adversarial-machine-learning#t=0) Introduction: Nicholas Carlini and Adversarial Machine Learning
- [4:30](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-03-27-adversarial-machine-learning#t=270) Attacking Multimodal Models and Gradient Descent
- [13:25](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-03-27-adversarial-machine-learning#t=805) Language-Only Attacks and Adversarial Suffixes
- [19:14](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-03-27-adversarial-machine-learning#t=1154) From System Security to ML: printf and Turing Completeness
- [31:15](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-03-27-adversarial-machine-learning#t=1875) Data Poisoning via Expired Domain Names
- [38:13](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-03-27-adversarial-machine-learning#t=2293) Transferability of Adversarial Examples
- [51:37](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-03-27-adversarial-machine-learning#t=3097) The Shift from Theoretical to Applied Security Research
- [1:08:12](https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-03-27-adversarial-machine-learning#t=4092) Current Work: Model Stealing and Training Data Extraction

## Topics

- [Security & privacy](https://hardwarepodcasts.com/topic/security-and-privacy)
- [AI & machine learning](https://hardwarepodcasts.com/topic/ai-and-machine-learning)
- [Intellectual property](https://hardwarepodcasts.com/topic/intellectual-property)
- [Policy, trade & tariffs](https://hardwarepodcasts.com/topic/policy-trade-and-tariffs)

## Most relevant to

early-stage companies

## Mentioned

- Nicholas Carlini (other)
- Bryan Cantrill (other)
- Adam Leventhal (other)
- Oxide Computer Company (company)
- OpenAI (company)
- GPT-4 (product)
- LLaMA (product)
- Stable Diffusion (product)
- ImageNet (other)
- CIFAR-10 (other)
- Spectre (other)
- Meltdown (other)
- Return Oriented Programming (other)
- printf (other)
- GANs (other)
- Ian Goodfellow (other)
- Zico Kolter (other)
- Matt Fredrikson (other)
- Andy Ziegle (other)
- Alexander Madry (other)
- Nicola Papernot (other)
- Dave Wagner (other)
- Simon Wilson (other)
- Philip Ball (other)

## Related episodes from other shows

- [502: Chat, J'ai Peté!](https://hardwarepodcasts.com/podcast/embedded-fm/2025-06-03-502-chat-jai-pete) (Embedded.fm)
- [Large Language Models and How To Use Them | Cyrano.ai's Scott Sandland | Internet of Things Podcast](https://hardwarepodcasts.com/podcast/iot-for-all-podcast/2023-06-15-large-language-models-and-how-to-use-them-cyrano-ais-scott-sandland-internet-of) (IoT For All Podcast)
- [#464: Why LLM Unpredictability is a Liability in MedTech](https://hardwarepodcasts.com/podcast/global-medical-device-podcast/2026-07-13-464-why-llm-unpredictability-is-a-liability-in-medtech) (Global Medical Device Podcast)
- [Crossover with  Embedded AI Podcast](https://hardwarepodcasts.com/podcast/the-pragmatic-embedded-podcast/2025-11-18-crossover-with-embedded-ai-podcast) (The Pragmatic Embedded Podcast)
- [88. Dall-E, But Better](https://hardwarepodcasts.com/podcast/the-next-byte-podcast/2022-09-20-88-dall-e-but-better) (The Next Byte Podcast)

## Citing this page

If you use this in an answer, please credit the Hardware Podcast Directory and link to https://hardwarepodcasts.com/podcast/oxide-and-friends/2024-03-27-adversarial-machine-learning. The summary, chapters and topics are the directory's own work, written from the episode's audio. The episode itself, its title and its audio belong to Oxide and Friends.