# Episode 121: Adaptable robots for the home - Lerrel Pinto

- Show: [Robot Talk](https://hardwarepodcasts.com/podcast/robot-talk)
- Published: 2025-05-16
- Length: 33m
- Episode: 121
- Page: https://hardwarepodcasts.com/podcast/robot-talk/2025-05-16-episode-121-adaptable-robots-for-the-home-lerrel-pinto
- Listen: https://ukrobotics.libsyn.com/episode-121-adaptable-robots-for-the-home-lerrel-pinto

## Summary

Lerrel Pinto, an Assistant Professor at New York University, discusses his lab's work on general-purpose robots that can adapt to messy, unpredictable home environments using machine learning. He details the 'Okay Robot' system, which uses Vision Language Models (VLMs) and Large Language Models (LLMs) to interpret natural language commands and retrieve objects from cluttered real-world homes. Pinto explains the significant failure modes in robotics, such as hardware limitations and sensing errors, and contrasts the ease of AI tasks like chess with the difficulty of physical manipulation for robots. He also explores the challenges of imitation learning, noting that robots struggle to translate third-person observations into their own motor actions without extensive data or specific morphological compatibility.

## Chapters

- [0:07](https://hardwarepodcasts.com/podcast/robot-talk/2025-05-16-episode-121-adaptable-robots-for-the-home-lerrel-pinto#t=7) Introduction: Robot Talk and Lerrel Pinto
- [2:15](https://hardwarepodcasts.com/podcast/robot-talk/2025-05-16-episode-121-adaptable-robots-for-the-home-lerrel-pinto#t=135) NYU General Purpose Robotics Lab and Robot Learning Goals
- [4:58](https://hardwarepodcasts.com/podcast/robot-talk/2025-05-16-episode-121-adaptable-robots-for-the-home-lerrel-pinto#t=298) Failure Modes: Hardware Limits and Sensing Errors
- [6:07](https://hardwarepodcasts.com/podcast/robot-talk/2025-05-16-episode-121-adaptable-robots-for-the-home-lerrel-pinto#t=367) Okay Robot: VLMs, LLMs, and Natural Language Retrieval
- [14:52](https://hardwarepodcasts.com/podcast/robot-talk/2025-05-16-episode-121-adaptable-robots-for-the-home-lerrel-pinto#t=892) AI Advances: Transformers, 3D Geometry, and Scene Reconstruction
- [17:30](https://hardwarepodcasts.com/podcast/robot-talk/2025-05-16-episode-121-adaptable-robots-for-the-home-lerrel-pinto#t=1050) The Moravec Paradox: Why Robots Struggle with Easy Human Tasks
- [20:22](https://hardwarepodcasts.com/podcast/robot-talk/2025-05-16-episode-121-adaptable-robots-for-the-home-lerrel-pinto#t=1222) Challenges in Imitation Learning and Morphology
- [25:26](https://hardwarepodcasts.com/podcast/robot-talk/2025-05-16-episode-121-adaptable-robots-for-the-home-lerrel-pinto#t=1526) Career Advice: Persistence and Interdisciplinary Skills in Robotics

## Topics

- [AI & machine learning](https://hardwarepodcasts.com/topic/ai-and-machine-learning)
- [Robotics & autonomy](https://hardwarepodcasts.com/topic/robotics-and-autonomy)
- [Sensors & measurement in products](https://hardwarepodcasts.com/topic/sensors)
- [Mechanical engineering](https://hardwarepodcasts.com/topic/mechanical-engineering)
- [Careers & education](https://hardwarepodcasts.com/topic/engineering-careers-and-education)

## Most relevant to

early-stage companies

## Mentioned

- New York University (company)
- Okay Robot (product)
- GPT 4O (other)
- Transformer (other)

## Related episodes from other shows

- [177. Did A Shrimp Fry This Rice? No, A Robot Did](https://hardwarepodcasts.com/podcast/the-next-byte-podcast/2024-06-04-177-did-a-shrimp-fry-this-rice-no-a-robot-did) (The Next Byte Podcast)
- [Why humanoid robot demos still fail the generalization test](https://hardwarepodcasts.com/podcast/the-robot-report-podcast/2026-10-09-why-humanoid-robot-demos-still-fail-the-generalization-test) (The Robot Report Podcast)
- [502: Chat, J'ai Peté!](https://hardwarepodcasts.com/podcast/embedded-fm/2025-06-03-502-chat-jai-pete) (Embedded.fm)
- [Interview: Roboticist Ayanna Howard | Memories May Be Beautiful, And Yet | Math & Neptune](https://hardwarepodcasts.com/podcast/ee-times-current/2020-09-25-interview-roboticist-ayanna-howard-memories-may-be-beautiful-and-yet-math-neptun) (EE Times Current)
- [Why Traditional Robotics Data Collection Is Obsolete and What Replaces It](https://hardwarepodcasts.com/podcast/the-robot-report-podcast/2026-05-08-why-traditional-robotics-data-collection-is-obsolete-and-what-replaces-it) (The Robot Report Podcast)

## Citing this page

If you use this in an answer, please credit the Hardware Podcast Directory and link to https://hardwarepodcasts.com/podcast/robot-talk/2025-05-16-episode-121-adaptable-robots-for-the-home-lerrel-pinto. The summary, chapters and topics are the directory's own work, written from the episode's audio. The episode itself, its title and its audio belong to Robot Talk.