Gm Global Technolgoy Operations, Inc.

Mantisvision Ltd.

Samsung Electronics Co., Ltd.

Michael Slutsky

Albert Shalumov

Ariel Lipson

Tzvi Philipp

Eyal Gordon

Itai Afek

Yonatan Samet

Daniel I Dobkin

Igal Bilik

Oded Bialer

Shahar Villeval

Dan Levi

Yun-Seok Lee

Michael Baltaxe

German Voronov

Eran Kishon

Daniel Kigli

Yehonatan Mandel

Lior Stein

Yuval Zelig

Ofer Geva

Avi Resler

Ariel Rubanenko

A method and system for correcting alignment of images received from cameras by estimating camera alignment parameters based on perturbative correction such that corrected alignment parameters satisfy pairwise epipolar constraints as well as global loop closure constraints. The corrected alignment parameters are estimated as a Least-Mean-Square solution to a system of linear equations and an estimation algorithm may be implemented as a part of a Random Sample Consensus criterion estimation for improved robustness. The method and system is fast and efficient and may be implemented in a real-time system.

Perturbative alignment correction for surround cameras

A system for occlusion reconstruction in surround views using temporal information is provided. The system includes an active camera device generating image data describing a first view of an operating environment and a computerized visual data controller. The controller includes programming to analyze the image data to generate a three-dimensional computerized representation of the operating environment, utilize the image data and the representation of the operating environment to synthesize a virtual camera view of the operating environment from a desired viewpoint, and identify an occlusion in the virtual camera view. The controller further includes programming to utilize historical iterations of the image data identify immobile objects within the operating environment and utilize the historical iterations to estimate filled information for the occlusion. The controller further includes programming to reconstruct the occlusion with the filled information using pixel data from the immobile objects and utilize the representation to provide navigational guidance.

System and method for occlusion reconstruction in surround views using temporal information

A disclosure is presented for estimating height of an object using ultrasonic sensors carried onboard a vehicle. The height may be estimated by generating ultrasonic distance measurements based on a time of flight associated with ultrasonic reflections detected with the ultrasonic sensors, generating a three-dimensional (3D) occupancy grid to volumetrically represent at least a portion of an environment within field of view of the ultrasonic sensors, the 3D occupancy grid including a plurality of volumetric pixels (voxels), assigning a count value to each of the voxels based on the distance measurements, and estimating the height according to a relative spatial relationship between the vehicle and the voxel having the count value with a greatest value.

Object height estimation using ultrasonic sensors

A visual perception system includes a scanning camera, color sensor with color filter array (CFA), and a classifier node. The camera captures full color pixel images of a target object, e.g., a traffic light, and processes the pixel images through a narrow band pass filter (BPF), such that the narrow BPF outputs monochromatic images of the target object. The color sensor and CFA receive the monochromatic images. The color sensor has at least three color channels each corresponding to different colors of spectral data in the monochromatic images. The classifier node uses a predetermined classification decision tree to classify constituent pixels of the monochromatic images into different color bins as a corresponding color of interest. The color of interest may be used to perform a control action, e.g., via an automated driver assist system (ADAS) control unit or an indicator device.

Generation of artificial color images from narrow spectral band data aboard a camera-equipped vehicle

Methods and system for training a neural network for depth estimation in a vehicle. The methods and systems receive respective training image data from at least two cameras. Fields of view of adjacent cameras of the at least two cameras partially overlap. The respective training image data is processed through a neural network providing depth data and semantic segmentation data as outputs. The neural network is trained based on a loss function. The loss function combines a plurality of loss terms including at least a semantic segmentation loss term and a panoramic loss term. The panoramic loss term includes a similarity measure regarding overlapping image patches of the respective image data that each correspond to a region of overlapping fields of view of the adjacent cameras. The semantic segmentation loss term quantifies a difference between ground truth semantic segmentation data and the semantic segmentation data output from the neural network.

Systems and methods for depth estimation in a vehicle

Systems and methods to perform noise-adaptive non-blind deblurring on an input image that includes blur and noise involve implementing a first neural network on the input image to obtain one or more parameters and performing regularized deconvolution to obtain a deblurred image from the input image. The regularized deconvolution uses the one or more parameters to control noise in the deblurred image. A method includes implementing a second neural network to remove artifacts from the deblurred image and provide an output image.

Noise-adaptive non-blind image deblurring

Systems and methods for generating a virtual view of a virtual camera based on an input image are described. A system for generating a virtual view of a virtual camera based on an input image can include a capturing device including a physical camera and a depth sensor. The system also includes a controller configured to determine an actual pose of the capturing device; determine a desired pose of the virtual camera for showing the virtual view; define an epipolar geometry between the actual pose of the capturing device and the desired pose of the virtual camera; and generate a virtual image depicting objects within the input image according to the desired pose of the virtual camera for the virtual camera based on an epipolar relation between the actual pose of the capturing device, the input image, and the desired pose of the virtual camera.

Generating virtual images based on captured image data

Systems and methods for generating a virtual view of a virtual camera based on an input scene are described. A capturing device typically includes a physical camera and a depth sensor and captures an input scene. A controller determines an actual pose of the capturing device and a desired pose of the virtual camera for showing the virtual view. The controller defines an epipolar geometry between the actual pose of the capturing device and the desired pose of the virtual camera. The controller generates an output image for the virtual camera based on an epipolar relation between the actual pose of the capturing device, the input scene, and the desired pose of the virtual camera.

Using epipolar reprojection for virtual view perspective change

Presented are intelligent vehicle systems with networked on-body vehicle cameras with camera-view augmentation capabilities, methods for making/using such systems, and vehicles equipped with such systems. A method for operating a motor vehicle includes a system controller receiving, from a network of vehicle-mounted cameras, camera image data containing a target object from a perspective of one or more cameras. The controller analyzes the camera image to identify characteristics of the target object and classify these characteristics to a corresponding model collection set associated with the type of target object. The controller then identifies a 3D object model assigned to the model collection set associated with the target object type. A new 'virtual' image is generated by replacing the target object with the 3D object model positioned in a new orientation. The controller commands a resident vehicle system to execute a control operation using the new image.

Intelligent vehicle systems and control logic for surround view augmentation with object model recognition

A vehicle control system for automated driver-assistance includes a low-resolution color camera that captures a color image of a field of view at a first resolution. The vehicle control system further includes a high-resolution scanning camera that captures a plurality of grayscale images, each of the grayscale images at a second resolution. The second resolution is higher than the first resolution. The grayscale images encompass the same field of view as the color image. The vehicle control system further includes one or more processors that perform a method that includes overlaying the plurality of grayscale images over the color image. The method further includes correcting motion distortion of one or more objects in the grayscale images. The method further includes generating a high-resolution output image by assigning color values to one or more pixels in the grayscale images based on the color image using a trained neural network.

Combining grayscale scanned images with color image to create high-resolution overlay images in vehicles

Systems and methods for projecting a multi-faceted image onto a convex polyhedron based on an input image are described. A system can include a controller configured to determine a mapping between pixels within a wide-angle image and a multi-faceted image, and generate the multi-faceted image based on the mapping.

Projecting multi-faceted image onto convex polyhedron based on wide-angle image

In various embodiments, methods and systems are provided for processing camera data from a camera system associated with a vehicle. In one embodiment, a method includes: storing a plurality of photorealistic scenes of an environment; training, by a processor, a machine learning model to produce a surround view approximating a ground truth surround view using the plurality of photorealistic scenes as training data; and processing, by a processor, the camera data from the camera system associated with the vehicle based on the trained machine learning model to produce a surround view of an environment of the vehicle.

Natural surround view

A vehicle, navigation system of the vehicle, and method for observing an environment of the vehicle. The navigation system includes a sensor and a processor. The sensor obtains a detection from an object located in an environment of the vehicle at a with-detection direction with respect to the sensor that includes the detection. The processor determines a no-detection direction that does not include the detection, assigns an empty space likelihood to an occupancy grid of the sensor along the no-detection direction, and generates a map of the environment using the occupancy grid.

Generalized 3D inverse sensor model

Methods and systems for training a non-blind deblurring module are disclosed. Unblurred test images and blurred test images are received, wherein each of the blurred test images is related to a corresponding one of the unblurred test images by a blur kernel term and a noise term. A regularized deconvolution sub-module and a convolutional neural network are jointly trained by adjusting a regularization parameter of a regularized deconvolution function and weights of a convolution neural network in order to minimize a cost function representative of a difference between each deblurred output image and a corresponding one of the unblurred test images.

Systems and methods for training a non-blind image deblurring module

A vehicle pose determining system and method for accurately estimating the pose of a vehicle (i.e., the location and/or orientation of a vehicle). The system and method use a form of sensor fusion, where output from vehicle dynamics sensors (e.g., accelerometers, gyroscopes, encoders, etc.) is used with output from vehicle radar sensors to improve the accuracy of the vehicle pose data. Uncorrected vehicle pose data derived from dynamics sensor data is compensated with correction data that is derived from occupancy grids that are based on radar sensor data. The occupancy grids, which are 2D or 3D mathematical objects that are somewhat like radar-based maps, must correspond to the same geographic location. The system and method use mathematical techniques (e.g., cost functions) to rotate and shift multiple occupancy grids until a best fit solution is determined, and the best fit solution is then used to derive the correction data that, in turn, improves the accuracy of the vehicle pose data.

Growing community of inventors

Kfar Saba, Israel

Michael Slutsky