Robot Talk · 16 May 2025 · 33m · Episode 121
Lerrel Pinto, an Assistant Professor at New York University, discusses his lab's work on general-purpose robots that can adapt to messy, unpredictable home environments using machine learning. He details the 'Okay Robot' system, which uses Vision Language Models (VLMs) and Large Language Models (LLMs) to interpret natural language commands and retrieve objects from cluttered real-world homes. Pinto explains the significant failure modes in robotics, such as hardware limitations and sensing errors, and contrasts the ease of AI tasks like chess with the difficulty of physical manipulation for robots. He also explores the challenges of imitation learning, noting that robots struggle to translate third-person observations into their own motor actions without extensive data or specific morphological compatibility.
Where this conversation describes AI being used to build or deliver a product. Part of the AI in practice tracker.
Engineering & simulationGPT-4O
The team used GPT-4O to label objects in images captured by the robot, creating a memory bank of described objects to enable the robot to locate items based on natural language queries.
“imagine you take an image, you ask GPT 4O. Um, what is in this image, and whatever it says is in this image. Imagine I sort of label that that description to those points in the image.”
Speaker 2 · Hear it at 10:30
Engineering & simulationTransformer architecture
The lab uses transformer architectures, typically found in large language models, to process robotic data by treating it like language, allowing the robot to learn behaviors from datasets.
“So the Transformer architecture is used in the large language models, but we often use them for a robotic problem. So we treat sort of robot data as, you know, if you squint at it, you know, you can say, oh, it's sort of like language.”
Speaker 2 · Hear it at 14:52
Engineering & simulation3D geometry learning tools
The system uses tools that reconstruct the room in 3D from visual observations to store object information in 3D locations within the robot's memory bank.
“Actually, okay robot uses this tool internally where as the robot is moving around using its visual observations, it reconstructs the scene, which is how its memory is stored, actually. So when you see an object, in your memory bank, you store information of that object, in a 3D location.”
Speaker 2 · Hear it at 14:52
early-stage companies
Other shows on the same ground, closest first.
Daniel and Forbode discuss Stanford University's 'Mobile Aloha' platform, a bimanual robot designed to learn household tasks through imitation learning rather than extensive simulation. They highlight the platform's ability to generalize from a static dataset of 825 operations and improve success rates by 90% after roughly 50 human-led demonstrations. The hosts emphasize the significance of reducing the cost of such robotic systems from hundreds of thousands to around $30,000, making general-purpose robotics more accessible for future commercial and domestic use.
Janine Sinanan-Singh, Director of Gen AI Research at Appen, explains why humanoid robot demos often fail the generalization test, emphasizing that high-quality egocentric video data and human judgment are critical for training models that can handle out-of-distribution tasks. She discusses the shift from classical robotics to AI-driven approaches, noting that while foundation models like Gemini offer zero-shot capabilities, latency and the need for intent understanding remain significant hurdles. The conversation also covers the role of synthetic data in scaling training, the importance of multi-agent systems for safety, and the economic pressures driving token optimization in AI development.
Elecia White and Christopher White discuss the TV adaptation of Martha Wells' 'Murderbot Diaries' and the deceptive nature of Large Language Models (LLMs) like ChatGPT, Claude, and Gemini. They highlight specific failures of AI, including lawyers citing non-existent legal precedents and GitHub Copilot agents generating endless, looping pull requests. The conversation also covers control systems, contrasting theoretical 'fancy math' like Kalman filters with empirical tuning, and mentions the Segway patent US7370713B1.
Georgia Tech professor and roboticist Ayanna Howard discusses the intersection of AI and robotics, highlighting how humans overtrust robots in high-risk scenarios and the challenges of generalizing robotic behavior across different environments. EE Times contributor Gary Hilson analyzes the semiconductor memory market, explaining why emerging memories like MRAM and 3D XPoint struggle to displace established DRAM and Flash due to cost and manufacturing hurdles. The episode also previews the upcoming Boards and Solutions Conference, focusing on embedded computer boards for industrial automation, smart cities, and edge computing.
Eric Chan, chief scientist and co-founder of Rhoda AI, explains the company's Direct Video Action (DVA) model, which uses internet video data to train robots for zero-shot learning. Chan contrasts this approach with traditional Vision-Language-Action (VLA) models, arguing that predicting future video frames provides a more scalable and data-efficient prior for robot control. The discussion covers the technical challenges of low-latency inference, the use of force/tactile inputs, and the future of prompting robots via language or single-shot demonstrations.
The summary, chapters and topics are generated from the episode’s audio by this directory, not written by Robot Talk. The episode itself belongs to its publisher.