X Development LLC

Google Inc.

Daniel Alex Lam

Eden Rephaeli

Sergey Levine

Yunfei Bai

Benjamin Holson

Seyed Mohammad Khansari Zadeh

Eric Jang

Guy Satat

Matthew Bennice

Andre Hentz

Jamie Gennis

Chelsea Finn

Daniel Kappler

Alexander Irpan

Brent Austin

Jiajun Xu

Nicolas Sievers

Implementations described herein relate to training and refining robotic control policies using imitation learning techniques. A robotic control policy can be initially trained based on human demonstrations of various robotic tasks. Further, the robotic control policy can be refined based on human interventions while a robot is performing a robotic task. In some implementations, the robotic control policy may determine whether the robot will fail in performance of the robotic task, and prompt a human to intervene in performance of the robotic task. In additional or alternative implementations, a representation of the sequence of actions can be visually rendered for presentation to the human can proactively intervene in performance of the robotic task.

System(s) and method(s) of using imitation learning in training and refining robotic control policies

Systems and methods are described include a robot and/or an associated computing system that can use various cues about an environment of the robot to apply a bias to increase the accuracy of speech transcription. In some implementations, audio data corresponding to a spoken instruction to a robot is received. Candidate transcriptions of the audio data are obtained. A respective action of the robot corresponding to each of the candidate transcriptions of the audio data is determined. One or more scores indicating characteristics of a potential outcome of performing the respective action corresponding to the candidate transcription of the audio data are determined for each of the candidate transcriptions of the audio data. A particular candidate transcription is selected from among the candidate transcriptions based at least on the one or more scores. The action determined for the particular candidate transcription is performed.

Speech recognition biasing

A system includes a robotic device, a sensor disposed on the robotic device, and circuitry configured to perform operations. The operations include determining a map that represents stationary features of an environment and receiving, from the sensor, sensor data representing the environment. The operations also include determining, based on the sensor data, a representation of an actor within the environment, where the representation includes keypoints representing corresponding body locations of the actor. The operations also include determining that a portion of a particular stationary feature is positioned within a threshold distance of a particular keypoint and, based on thereon, updating the map to indicate that the portion is to be cleaned. The operations further include, based on the map as updated, causing the robotic device to clean the portion of the particular stationary feature.

Monitoring of surface touch points for precision cleaning

Methods, systems, and apparatus, including computer programs stored on computer storage devices, for multi-tiered command processing are disclosed. One of the methods includes operating an audio processing system to detect voice commands in a set of first commands controlling behavior of the robot and in a set of second commands controlling behavior of the robot, wherein the system detects commands in the set of first commands using a local recognition model that does not require communication over a network, and detects commands in the set of second commands with a server over a network; executing one or more applications to respond to commands in a set of application commands that include commands different from the first commands and second commands; and detecting and carrying out commands, according to a predetermined hierarchy that prioritizes detection of the first commands first, the second commands second, and the application commands third.

Multi-tiered command processing

Methods, apparatus, systems, and computer-readable media are provided for creating, storing, and/or offloading tagged robot sensor data. In various implementations, a first plurality of sensor data points that are sampled by one or more sensors associated with a robot and that share a first attribute may be identified. Each of the first plurality of sensor data points may be tagged with a first tag, which may be indicative of the first attribute. A context in which a robot is operating may be identified. A first transport rule that governs how sensor data points tagged with the first tag are treated when the robot operates in the context may then be identified. At least a subset of the first plurality of tagged sensor data points may then be offloaded from the robot and/or stored locally on the robot pursuant to the first transport rule.

Tagged robot sensor data

Growing community of inventors

San Francisco, CA, United States of America

Daniel Alex Lam