X Development LLC

Google Inc.

Gm Cruise Holdings LLC

Guy Satat

Michael Joseph Quinlan

Sean Kirmani

Eden Rephaeli

Anelia Angelova

Benjamin Holson

Daniel Alex Lam

Ariel Gordon

Jiajun Xu

Daniel Lam

Systems and techniques are provided for fusing raw sensor data captured by a camera sensor and one or more depth sensors to generate a dense depth map. An example process includes receiving raw camera data captured by a camera sensor and descriptive of a scene, receiving raw depth data captured by one or more depth-sensing sensors and descriptive of the scene, and providing the raw camera data and the raw depth data to a neural network, which is configured to fuse the raw camera data and the raw depth data. The example process can further include generating a depth map of the scene based on the fusion of the raw camera data and the raw depth data.

Raw sensor data fusion between a camera sensor and a depth sensor

The present disclosure generally relates to determining depth information for an image frame and, more specifically, to using partially overlapping image portions to propagate depth information through the entire image frame. In some aspects, a method of the disclosed technology includes steps for identifying an overlapping portion of a first image frame and a second image frame, and a non-overlapping portion of the first image frame; determining depth information for one or more pixels in the overlapping portion of the first image frame, wherein a stereo overlap technique is used to determine the depth information; determining semantic information for the non-overlapping portion of the first image frame; and providing the depth information and the semantic information to a machine learning model to get depth information for the non-overlapping portion of first image frame. Systems and machine-readable media are also provided.

Depth from vision with partial stereo overlap

A method includes receiving, from a multiscopic image capture system, a plurality of images depicting a scene. The method includes determining, by application of a neural network based on the plurality of images, a disparity map of the scene. The neural network includes a plurality of layers, and the layers include a rectification layer. The method include determining a matching error of the disparity map based on differences between corresponding pixels of two or more images associated with the disparity map. The method includes back-propagating the matching error to the rectification layer of the neural network. Back-propagating the matching error includes updating one or more weights applied to the rectification layer.

Systems and methods for automatically calibrating multiscopic image capture systems

A method includes receiving a first depth map that includes a plurality of first pixel depths and a second depth map that includes a plurality of second pixel depths. The first depth map corresponds to a reference depth scale and the second depth map corresponds to a relative depth scale. The method includes aligning the second pixel depths with the first pixel depths. The method includes transforming the aligned region of the second pixel depths such that transformed second edge pixel depths of the aligned region are coextensive with first edge pixel depths surrounding the corresponding region of the first pixel depths. The method includes generating a third depth map. The third depth map includes a first region corresponding to the first pixel depths and a second region corresponding to the transformed and aligned region of the second pixel depths.

Fusing multiple depth sensing modalities

A method includes receiving sensor data representing a first object in an environment and generating, based on the sensor data, a first state vector that represents physical properties of the first object. The method also includes generating, by a first machine learning model and based on the first state vector and a second state vector that represents physical properties of a second object previously observed in the environment, a metric indicating a likelihood that the first object is the same as the second object. The method further includes determining, based on the metric, to update the second state vector and updating, by a second machine learning model configured to maintain the second state vector over time and based on the first state vector, the second state vector to incorporate into the second state vector information concerning physical properties of the second object as represented in the first state vector.

Object association using machine learning models

A method includes identifying a target surface in an environment of a robotic device. The method further includes controlling a moveable component of the robotic device to move along a motion path relative to the target surface, wherein the moveable component comprises a light source and a camera. The method additionally includes receiving a plurality of images from the camera when the moveable component is at a plurality of poses along the motion path and when the light source is illuminating the target surface. The method also includes determining bidirectional reflectance distribution function (BRDF) image data, wherein the BRDF image data comprises the plurality of images converted to angular space with respect to the target surface. The method further includes determining, based on the BRDF image data and by applying at least one pre-trained machine learning model, a material property of the target surface.

Recovering material properties with active illumination and camera on a robot manipulator

A system includes a robotic device, a sensor disposed on the robotic device, and circuitry configured to perform operations. The operations include determining a map that represents stationary features of an environment and receiving, from the sensor, sensor data representing the environment. The operations also include determining, based on the sensor data, a representation of an actor within the environment, where the representation includes keypoints representing corresponding body locations of the actor. The operations also include determining that a portion of a particular stationary feature is positioned within a threshold distance of a particular keypoint and, based on thereon, updating the map to indicate that the portion is to be cleaned. The operations further include, based on the map as updated, causing the robotic device to clean the portion of the particular stationary feature.

Monitoring of surface touch points for precision cleaning

A method includes receiving sensor data from a plurality of robot sensors on a robot. The method includes generating a depth map that includes a plurality of pixel depths. The method includes determining, for each respective pixel depth, based on the at least one robot sensor associated with the respective pixel depth, a pixelwise confidence level indicative of a likelihood that the respective pixel depth accurately represents a distance between the robot and a feature of the environment. The method includes generating a pixelwise filterable depth map for a control system of the robot. The pixelwise filterable depth map is filterable to produce a robot operation specific depth map. The robot operation specific depth map is determined based on a comparison of each respective pixelwise confidence level with a confidence threshold corresponding to at least one operation of the robot controlled by the control system of the robot.

Growing community of inventors

Sunnyvale, CA, United States of America

Guy Satat