Waymo LLC

Comcast Cable Communications, LLC

International Business Machines Corporation

Ruichi Yu

Hongcheng Wang

Xu Chen

Congcong Li

Alper Ayvaci

Hang Zhao

Rui Min

Kang Li

Hong Li

Yu-Han Chen

Chen Wu

Jonathon Shlens

Ian James Mahon

Henrik Kretzschmar

Shiwei Sheng

Noha Waheed Ahmed Radwan

Sachithra Madhawa Hemachandra

Yu Ouyang

Ching-Yung Lin

Jui-Hsin Lai

Chen Sun

Junhua Mao

Kevin Patrick Murphy

Jiyang Gao

Chun-Fu Chen

Maya Kabkab

Nichola Abdo

Ruizhongtai Qi

Guangnan Ye

Dillon Cower

Tao Han

Robert Cosgriff

Longlong Jing

Yingwei Li

Luming Tang

Chieh-En Tsai

Chang Gao

Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for generating a distance estimate for a target object that is depicted in an image of a scene in an environment. The system obtains data specifying (i) a target portion of the image that depicts the target object detected in the image, and (ii) one or more reference portions of the image that each depict a respective reference object detected in the image. The system further obtains, for each of the one or more reference objects, a respective distance measurement for the reference object that is a measurement of a distance from the reference object to a specified location in the environment. The system processes the obtained data to generate a distance estimate for the target object that is an estimate of a distance from the target object to the specified location in the environment.

Long range distance estimation using reference objects

Methods, systems, and/or apparatuses are described for detecting relevant motion of objects of interest (e.g., persons and vehicles) in surveillance videos. As described herein input data based on a plurality of captured images and/or video is received. The input data may then be pre-processed and used as an input into a convolution network that may, in some instances, have elements that perform both spatial-wise max pooling and temporal-wise max pooling. The convolution network may be used to generate a plurality of prediction results of relevant motion of the objects of interest.

Relevant motion detection in video

The described aspects and implementations enable fast and accurate object identification in autonomous vehicle (AV) applications by combining radar data with camera images. In one implementation, disclosed is a method and a system to perform the method that includes obtaining a radar image of a first hypothetical object in an environment of the AV, obtaining a camera image of a second hypothetical object in the environment of the AV, and processing the radar image and the camera image using one or more machine-learning models MLMs to obtain a prediction measure representing a likelihood that the first hypothetical object and the second hypothetical object correspond to a same object in the environment of the AV.

Association of camera images and radar data in autonomous vehicle applications

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting three-dimensional object locations from images. One of the methods includes obtaining a sequence of images that comprises, at each of a plurality of time steps, a respective image that was captured by a camera at the time step; generating, for each image in the sequence, respective pseudo-lidar features of a respective pseudo-lidar representation of a region in the image that has been determined to depict a first object; generating, for a particular image at a particular time step in the sequence, image patch features of the region in the particular image that has been determined to depict the first object; and generating, from the respective pseudo-lidar features and the image patch features, a prediction that characterizes a location of the first object in a three-dimensional coordinate system at the particular time step in the sequence.

Three-dimensional location prediction from images

The described aspects and implementations enable fast and accurate verification of radar detection of objects in autonomous vehicle (AV) applications using combined processing of radar data and camera images. In one implementation, disclosed is a method and a system to perform the method that includes obtaining a radar data characterizing intensity of radar reflections from an environment of the AV, identifying, based on the radar data, a candidate object, obtaining a camera image depicting a region where the candidate object is located, and processing the radar data and the camera image using one or more machine-learning models to obtain a classification measure representing a likelihood that the candidate object is a real object.

Identification of spurious radar detections in autonomous vehicle applications

The described aspects and implementations enable efficient calibration of a sensing system of a vehicle. In one implementation, disclosed is a method and a system to perform the method, the system including the sensing system configured to obtain a plurality of images associated with a corresponding time of a plurality of times. The system further includes a data processing system operatively coupled to the sensing system and configured to generate a plurality of sets of feature tensors (FTs) associated with one or more objects of the environment depicted in a respective image. The data processing system is further to obtain a combined FT and process the combined FT using a neural network to identify one or more tracks characterizing motion of a respective object.

End-to-end object tracking using neural networks with attention

A camera system may automatically identify high frequency motion regions in a field of view of the camera system. Analysis of motion events detected at different regions of a scene in the field of view, may be performed. Similar events frequently occurring at a certain region may be recorded/accumulated and compared to a threshold. Regions with high frequency activity may be determined and a user may be notified. If the user is not interested in events within the high activity regions, the regions may be ignored and notifications to the user may cease. The camera system may determine specific actions/events occurring in a region, such as a person approaching or leaving a home.

Methods and systems for determining object activity within a region of interest

Aspects of the disclosure relate to controlling a vehicle. For instance, using a camera, a first camera image including a first object may be captured. A first bounding box for the first object and a distance to the first object may be identified. A second camera image including a second object may be captured. A second bounding box for the second image and a distance to the second object may be identified. Whether the first object is the second object may be determined using a plurality of models to compare visual similarity of the two bounding boxes, to compare a three-dimensional location based on the distance to the first object and a three-dimensional location based on the distance to the second object, and to compare results from the first and second models. The vehicle may be controlled in an autonomous driving mode based on a result of the third model.

Long-range object detection, localization, tracking and classification for autonomous vehicles

Example embodiments relate to techniques for three dimensional (3D) object detection and localization. A computing system may cause a radar unit to transmit radar signals and receive radar reflections relative to an environment of a vehicle. Based on the radar reflections, the computing system may determine a heading and a range for a nearby object. The computing system may also receive an image depicting a portion of the environment that includes the object from a vehicle camera and remove peripheral areas of the image to generate an image patch that focuses upon the object based on the heading and the range for the object. The image patch and the heading and the range for the object can be provided as inputs into a neural network that provides output parameters corresponding to the object, which can be used to control the vehicle.

Methods and systems for three dimensional object detection and localization

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for associating a new measurement of an object surrounding a vehicle with a maintained track. One of the methods includes receiving an object track for a particular object, receiving a new measurement characterizing a new object at a new time step, and determining whether the new object is the same as the particular object, comprising: generating a representation of the new object at the new and preceding time steps; generating a representation of the particular object at the new and preceding time steps; processing a first network input comprising the representations using a first neural network to generate an embedding of the first network input; and processing the embedding of the first network input using a second neural network to generate a predicted likelihood that the new object and the particular object are the same.

Geo-motion and appearance aware data association

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating object interaction predictions using a neural network. One of the methods includes obtaining a sensor input derived from data generated by one or more sensors that characterizes a scene. The sensor input is provided to an object interaction neural network. The object interaction neural network is configured to process the sensor input to generate a plurality of object interaction outputs. Each respective object interaction output includes main object information and interacting object information. The respective object interaction outputs corresponding to the plurality of regions in the sensor input are received as output of the object interaction neural network.

Interacted object detection neural network

Methods, systems, and/or apparatuses are described for detecting relevant motion of objects of interest (e.g., persons and vehicles) in surveillance videos. As described herein input data based on a plurality of captured images and/or video is received. The input data may then be pre-processed and used as an input into a convolution network that may, in some instances, have elements that perform both spatial-wise max pooling and temporal-wise max pooling. Based on The convolution network may be used to generate a plurality of prediction results of relevant motion of the objects of interest.

Growing community of inventors

Mountain View, CA, United States of America

Ruichi Yu