Waymo LLC

Google Inc.

Other

Yale University

X Development LLC

Abhijit S Ogale

Wan-Yen Lo

David Ian Franklin Ferguson

Alexander Krizhevsky

David Ian Ferguson

David Harrison Silver

Dave Ferguson

Ehud Benyamin Rivlin

Dmitri A Dolgov

Andrew Hughes Chatham

Vadim Furman

Nathaniel Fairfield

Chen Wu

Rodrigo Lima Carceroni

Congcong Li

Alper Ayvaci

Yangli Hector Yee

Ehud Rivlin

Hyman Murveit

Pankaj Chaudhari

Carl Kershaw

Augusto Roman

Henrik Kretzschmar

Yu-Han Chen

Matthew Wang

Owen Brydon

Victoria Dean

Jiajun Zhu

Dragomir Dimitrov Anguelov

Jay Yagnik

Carole Dulong

Luc Vincent

Andreas Wendel

Andrea Lynn Frome

Scott Ettinger

Mayank Bansal

Stephane Lafon

Edward Hsiao

Maxim Krivokon

Yang Gao

Dimitrios Lymberopoulos

Jeremy Bryant Pack

Andreas Savvides

Yiannis Aloimonos

Methods and systems for the use of detected objects for image processing are described. A computing device autonomously controlling a vehicle may receive images of the environment surrounding the vehicle from an image-capture device coupled to the vehicle. In order to process the images, the computing device may receive information indicating characteristics of objects in the images from one or more sources coupled to the vehicle. Examples of sources may include RADAR, LIDAR, a map, sensors, a global positioning system (GPS), or other cameras. The computing device may use the information indicating characteristics of the objects to process received images, including determining the approximate locations of objects within the images. Further, while processing the image, the computing device may use information from sources to determine portions of the image to focus upon that may allow the computing device to determine a control strategy based on portions of the image.

Use of detected objects for image processing

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for selecting locations in an environment of a vehicle where objects are likely centered and determining properties of those objects. One of the methods includes receiving an input characterizing an environment external to a vehicle. For each of a plurality of locations in the environment, a respective first object score that represents a likelihood that a center of an object is located at the location is determined. Based on the first object scores, one or more locations from the plurality of locations are selected as locations in the environment at which respective objects are likely centered. Object properties of the objects that are likely centered at the selected locations are also determined.

Neural networks for object detection and characterization

A vehicle may receive one or more images of an environment of the vehicle. The vehicle may also receive a map of the environment. The vehicle may also match at least one feature in the one or more images with corresponding one or more features in the map. The vehicle may also identify a given area in the one or more images that corresponds to a a portion of the map that is within a threshold distance to the one or more features. The vehicle may also compress the one or more images to include a lower amount of details in areas of the one or more images other than the given area. The vehicle may also provide the compressed images to a remote system, and responsively receive operation instructions from the remote system.

Image and video compression for remote vehicle assistance

An autonomous vehicle is configured to detect an active turn signal indicator on another vehicle. An image-capture device of the autonomous vehicle captures an image of a field of view of the autonomous vehicle. The autonomous vehicle captures the image with a short exposure to emphasize objects having brightness above a threshold. Additionally, a bounding area for a second vehicle located within the image is determined. The autonomous vehicle identifies a group of pixels within the bounding area based on a first color of the group of pixels. The autonomous vehicle also calculates an oscillation of an intensity of the group of pixels. Based on the oscillation of the intensity, the autonomous vehicle determines a likelihood that the second vehicle has a first active turn signal. Additionally, the autonomous vehicle is controlled based at least on the likelihood that the second vehicle has a first active turn signal.

Vision-based indicator signal detection using spatiotemporal filtering

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating object predictions using a neural network. One of the methods includes receiving respective projections of a plurality of channels of input sensor data, wherein each channel of input sensor data represents different respective characteristics of electromagnetic radiation reflected off of one or more objects. Each of the projections of the plurality of channels of input sensor data are provided to a neural network subsystem trained to receive projections of input sensor data as input and to provide an object prediction as an output. At the output of the neural network subsystem, an object prediction that predicts a region of space that is likely to be occupied by an object is received.

Object detection neural network

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a future occupancy prediction for a region of an environment. In one aspect, a method comprises: receiving sensor data generated by a sensor system of a vehicle that characterizes an environment in a vicinity of the vehicle as of a current time point, wherein the sensor data comprises a plurality of sensor samples characterizing the environment that were each captured at different time points; processing a network input comprising the sensor data using a neural network to generate an occupancy prediction output for a region of the environment, wherein: the occupancy prediction output characterizes, for one or more future intervals of time after the current time point, a respective likelihood that the region of the environment will be occupied by an agent in the environment during the future interval of time.

Occupancy prediction neural networks

Methods and system are provided for training and using a model to determine states of lanes of interest. For instance, image data including an image and an associated label identifying at least one traffic light, a state of the at least one traffic light, and a lane controlled by the at least one traffic light are received and used to train the model such that the model is configured to, in response to receiving an image and a lane of interest included in the image, output a lane state for the lane of interest. This model is then used by a vehicle in order to determine a state of a lane of interest. This state is then used to control the vehicle in an autonomous driving mode based on the state of the lane of interest.

Traffic light detection and lane state recognition for autonomous vehicles

Aspects of the disclosure relate to training and using a phrase recognition model to identify phrases in images. As an example, a selected phrase list may include a plurality of phrases is received. Each phrase of the plurality of phrases includes text. An initial plurality of images may be received. A training image set may be selected from the initial plurality of images by identifying the phrase-containing images that include one or more phrases from the selected phrase list. Each given phrase-containing image of the training image set may be labeled with information identifying the one or more phrases from the selected phrase list included in the given phrase-containing images. The model may be trained based on the training image set such that the model is configured to, in response to receiving an input image, output data indicating whether a phrase of the plurality of phrases is included in the input image.

Phrase recognition model for autonomous vehicles

A neural network system for identifying positions of objects in an input image can include an object detector neural network, a memory interface subsystem, and an external memory. The object detector neural network is configured to, at each time step of multiple successive time steps, (i) receive a first neural network input that represents the input image and a second neural network input that identifies a first set of positions of the input image that have each been classified as showing a respective object of the set of objects, and (ii) process the first and second inputs to generate a set of output scores that each represents a respective likelihood that an object that is not one of the objects shown at any of the positions in the first set of positions is shown at a respective position of the input image that corresponds to the output score.

Neural networks for object detection

A vehicle may receive one or more images of an environment of the vehicle. The vehicle may also receive a map of the environment. The vehicle may also match at least one feature in the one or more images with corresponding one or more features in the map. The vehicle may also identify a given area in the one or more images that corresponds to a portion of the map that is within a threshold distance to the one or more features. The vehicle may also compress the one or more images to include a lower amount of details in areas of the one or more images other than the given area. The vehicle may also provide the compressed images to a remote system, and responsively receive operation instructions from the remote system.

A computing device of a first vehicle may receive a first image and a second image of a second vehicle having flashing light signals. The computing device may determine, in the first image and the second image, an image region that bounds the second vehicle such that the image region substantially encompasses the second vehicle. The computing device may determine a polar grid that partitions the image region in the first image and the second image into polar bins, and identify portions of image data exhibiting a change in color and a change in brightness between the first image and the second image. The computing device may determine a type of the flashing light signals and a type of the second vehicle; and accordingly provide instructions to control the first vehicle.

Vehicle control using vision-based flashing light signal detection

Systems, methods, devices, and other techniques for training a trajectory planning neural network system to determine waypoints for trajectories of vehicles. A neural network training system can train the trajectory planning neural network system on the multiple training data sets. Each training data set can include: (i) a first training input that characterizes a set of waypoints that represent respective locations of a vehicle at each of a series of first time steps, (ii) a second training input that characterizes at least one of (a) environmental data that represents a current state of an environment of the vehicle or (b) navigation data that represents a planned navigation route for the vehicle, and (iii) a target output characterizing a waypoint that represents a target location of the vehicle at a second time step that follows the series of first time steps.

Neural networks for vehicle trajectory planning

Growing community of inventors

Sunnyvale, CA, United States of America

Abhijit S Ogale