Episode 74: Teaching robots to grasp - Edward Johns

Engineering & simulationDali Bot (image generation models)

The team used AI image generators to create realistic goal images for robot manipulation tasks, allowing robots to understand object placement without manual rule definition.

“So we're exploiting the fact that there's lots of this data available in terms of images and language, but this data wasn't curated specifically for robotics. We're now thinking of creative ways to use that for robotics. And we found that this works really well to allow robots to imagine these goal images that they might then want to create over the next few seconds.”

Engineering & simulation

The team manually filtered AI-generated images to remove 'arty' or unrealistic outputs, retaining only realistic scenes to ensure the robot could physically recreate the generated goals.

“So we did that manually. We we did things like we checked the number of objects in the generated image and made sure that it matched up with the number of objects in the in the real scene. Um, because for example, one of the examples that we might have seen would be, There would be 10 knives, uh, generated in the scene, even though we asked for only one knife…”

Engineering & simulation

The lab uses imitation learning where humans physically move the robot's arms to teach it tasks, combining visual observations with actions to learn a control policy that predicts optimal actions.

“So rather than doing that, what we do is the other approach, which is where we physically hold the robot and actually move that robot physically to show it how to perform a task. And this means that the body that the robot is observing is just its own body, and therefore it doesn't need to map that to anybody else's body…”