Ford Global Technolgoies, LLC

Michigan State University

Praveen Narayanan

Punarjay Chakravarty

Gaurav Pandey

Francois Charette

Lisa Scaria

Ryan Burke

Nikita Jaipuria

Kaushik Balakrishnan

Sarah Houts

Graham Mills

Vijay K Nagasamy

Marcos Paul Gerardo Castro

Xiaoming Liu

Apurbaa Mallik

Daniel Morris

Ganesh Kumar

Tyler Reid

Tom Roussel

Yunfei Long

Vidya Nariyambut Murali

Jinesh J Jain

Scott Vincent Myers

Harpreetsingh Banvait

Ashley Elizabeth Micks

Alexandru Mihai Gurghian

Shreyasha Paudel

Gurjeet Singh

Mark Crawford

Mohsen Lakehal-ayat

Ramchandra Ganesh Karandikar

Sowndarya Sundar

Rohun Atluri

David Silver

A computer includes a processor and a memory storing instructions executable by the processor to receive radar data including a radar pixel having a radial velocity from a radar; receive camera data including an image frame including camera pixels from a camera; map the radar pixel to the image frame; generate a region of the image frame surrounding the radar pixel; determine association scores for the respective camera pixels in the region; select a first camera pixel of the camera pixels from the region, the first camera pixel having a greatest association score of the association scores; and calculate a full velocity of the radar pixel using the radial velocity of the radar pixel and a first optical flow at the first camera pixel. The association scores indicate a likelihood that the respective camera pixels correspond to a same point in an environment as the radar pixel.

Vehicle using full-velocity determination with radar

A computer includes a processor and a memory storing instructions executable by the processor to receive radar data from a radar, the radar data including radar pixels having respective measured depths; receive camera data from a camera, the camera data including an image frame including camera pixels; map the radar pixels to the image frame; generate respective regions of the image frame surrounding the respective radar pixels; for each region, determine confidence scores for the respective camera pixels in that region; output a depth map of projected depths for the respective camera pixels based on the confidence scores; and operate a vehicle including the radar and the camera based on the depth map. The confidence scores indicate confidence in applying the measured depth of the radar pixel for that region to the respective camera pixels.

Depth map generation

A first image can be acquired from a first sensor included in a vehicle and input to a deep neural network to determine a first bounding box for a first object. A second image can be acquired from the first sensor. Input latitudinal and longitudinal motion data from second sensors included in the vehicle corresponding to the time between inputting the first image and inputting the second image. A second bounding box can be determined by translating the first bounding box based on the latitudinal and longitudinal motion data. The second image can be cropped based on the second bounding box. The cropped second image can be input to the deep neural network to detect a second object. The first image, the first bounding box, the second image, and the second bounding box can be output.

Image annotation for deep neural networks

A system includes a computer including a processor and a memory, the memory storing instructions executable by the processor to generate a synthetic image by adjusting respective color values of one or more pixels of a reference image based on a specified meteorological optical range from a vehicle sensor to simulated fog, and input the synthetic image to a machine learning program to train the machine learning program to identify a meteorological optical range from the vehicle sensor to actual fog.

Enhanced vehicle operation

A computer, including a processor and a memory, the memory including instructions to be executed by the processor to input a fisheye image to a vector quantized variational autoencoder. The vector quantized variational autoencoder can encode the fisheye image to first latent variables based on an encoder. The vector quantized variational autoencoder can quantize the first latent variables to generate second latent variables based on a dictionary of embeddings. The vector quantized variational autoencoder can decode the second latent variables to a rectified rectilinear image using a decoder and output the rectified rectilinear image.

Image rectification

Example localization systems and methods are described. In one implementation, a method receives a camera image from a vehicle camera and cleans the camera image using a VAE-GAN (variational autoencoder combined with a generative adversarial network) algorithm. The method further receives a vector map related to an area proximate the vehicle and generates a synthetic image based on the vector map. The method then localizes the vehicle based on the cleaned camera image and the synthetic image.

Localization systems and methods

The present disclosure discloses a system and a method that includes receiving, at a decoder, a latent representation of an image having a first domain, and generating a reconstructed image having a second domain, wherein the reconstructed image is generated based on the latent representation.

Domain translation network for performing image translation

A method for training an autonomous vehicle to reach a target location. The method includes detecting the state of an autonomous vehicle in a simulated environment, and using a neural network to navigate the vehicle from an initial location to a target destination. During the training phase, a second neural network may reward the first neural network for a desired action taken by the autonomous vehicle, and may penalize the first neural network for an undesired action taken by the autonomous vehicle. A corresponding system and computer program product are also disclosed and claimed herein.

Automatic navigation using deep reinforcement learning

An end-to-end deep-learning-based system that can solve both ASR and TTS problems jointly using unpaired text and audio samples is disclosed herein. An adversarially-trained approach is used to generate a more robust independent TTS neural network and an ASR neural network that can be deployed individually or simultaneously. The process for training the neural networks includes generating an audio sample from a text sample using the TTS neural network, then feeding the generated audio sample into the ASR neural network to regenerate the text. The difference between the regenerated text and the original text is used as a first loss for training the neural networks. A similar process is used for an audio sample. The difference between the regenerated audio and the original audio is used as a second loss. Text and audio discriminators are similarly used on the output of the neural network to generate additional losses for training.

Joint automatic speech recognition and text to speech conversion using adversarial neural networks

The present disclosure discloses a system and a method. In an example implementation, the system and the method generate, at a first encoder neural network, an encoded representation of image features of an image received from a vehicle sensor of a vehicle. The system and method can also generate, at a second encoder neural network, an encoded representation of map tile features and generate, at the decoder neural network, a semantically segmented map tile based on the encoded representation of image features, the encoded representation of map tile features, and Global Positioning System (GPS) coordinates of the vehicle. The semantically segmented map tile includes a location of the vehicle and detected objects depicted within the image with respect to the vehicle.

Neural network for localization and object detection

Various examples of hybrid metric-topological camera-based localization are described. A single image sensor captures an input image of an environment. The input image is localized to one of a plurality of topological nodes of a hybrid simultaneous localization and mapping (SLAM) metric-topological map which describes the environment as the plurality of topological nodes at a plurality of discrete locations in the environment. A metric pose of the image sensor can be determined using a Perspective-n-Point (PnP) projection algorithm. A convolutional neural network (CNN) can be trained to localize the input image to one of the plurality of topological nodes and a direction of traversal through the environment.

Hybrid metric-topological camera-based localization

A speech conversion system is described that includes a hierarchical encoder and a decoder. The system may comprise a processor and memory storing instructions executable by the processor. The instructions may comprise to: using a second recurrent neural network (RNN) (GRU1) and a first set of encoder vectors derived from a spectrogram as input to the second RNN, determine a second concatenated sequence; determine a second set of encoder vectors by doubling a stack height and halving a length of the second concatenated sequence; using the second set of encoder vectors, determine a third set of encoder vectors; and decode the third set of encoder vectors using an attention block.

Hierarchical encoder for speech conversion system

A system comprises a computer that includes a processor and a memory. The memory stores instructions executable by the processor to estimate a first joint probability distribution of first data with respect to second data based on first and second marginal probability distributions of the first and second data, wherein the first data is from a first sensor based on a first sensor coordinate system, and the second data is from a second sensor based on a second sensor coordinate system, to estimate a second joint probability distribution of the first data with respect to the second data based on a projection of the first data onto a plane defined in the second sensor coordinate system, to estimate a rigid transformation between the first sensor coordinate system and the second sensor coordinate system by minimizing a distance between the first and second joint probability distributions, wherein the second joint probability distribution is a function of a set of extrinsic calibration parameters, and based on the set of the extrinsic calibration parameters, to detect an object in the first and second data.

Sensor fusion

Systems, methods, and devices for determining a location of a vehicle or other device are disclosed. A method includes receiving sensor data from a sensor and determining a prior map comprising LIDAR intensity values. The method includes extracting a sub-region of the prior map around a hypothesis position of the sensor. The method includes extracting a Gaussian Mixture Model (GMM) distribution of intensity values for a region of the sensor data by expectation-maximization and calculating a log-likelihood for the sub-region of the prior map based on the GMM distribution of intensity values for the sensor data.

Localization for autonomous vehicles using gaussian mixture models

A computer, including a processor and a memory, the memory including instructions to be executed by the processor to generate two or more stereo pairs of synthetic images and generate two or more stereo pairs of real images based on the two or more stereo pairs of synthetic images using a generative adversarial network (GAN), wherein the GAN is trained using a six-axis degree of freedom (DoF) pose determined based on the two or more pairs of real images. The instructions can further include instructions to train a deep neural network based on a sequence of real images and operate a vehicle using the deep neural network to process a sequence of video images acquired by a vehicle sensor.

Growing community of inventors

San Jose, CA, United States of America

Praveen Narayanan