Nvidia Corporation

Intel Corporation

Niranjan Avadhanam

Yuzhuo Ren

Sakthivel Sivaraman

Shagan Sah

Nishant Puri

Hairong Jiang

Nuri Murat Arar

Atousa Torabi

Rajath Shetty

Feng Hu

Sujay Yadawadkar

Jason Conrad Roche

Siyue Wu

Pavlo Molchanov

Dawid Stanislaw Pajak

Ambrish Dantrey

Sumit Bhattacharya

Ram Ganapathi

Revanth Reddy Nalla

Abhijit Patait

Sumit Kumar Bhattacharya

Thrinadh Kottana

Phani Kumar Nyshadham

Varun Allagadapa

Anshul Jain

Ratin Kumar

Shalini Gupta

Shubhadeep Das

Eric Viscito

Prashant Sohani

Ravi Kumar Boddeti

Simon John Baker

Guangli Dai

Nuri Arar Murat

Shivakumar D R

Sreekanth Nakkala

Ankit Pashiney

In various examples, color harmonization is applied to images of an environment in a reference light space. For example, different cameras on an ego-object may use independent capturing algorithms to generate processed images of the environment representing a common time slice using different capture configuration parameters. The processed images may be transformed into deprocessed images by inverting one or more stages of image processing to transform the processed images into a reference light space of linear light, and color harmonization may be applied to the deprocessed images in the reference light space. After applying color harmonization, corresponding image processing may be reapplied to the harmonized images using corresponding capture configuration parameters, the resulting processed harmonized images may be stitched into a stitched image, and a visualization of the stitched image may be presented (e.g., on a monitor visible to an occupant or operator of the ego-object).

Image stitching with color harmonization of de-processed images for surround view systems and applications

In various examples, the present disclosure relates to using temporal filters for automated real-time classification. The technology described herein improves the performance of a multiclass classifier that may be used to classify a temporal sequence of input signals-such as input signals representative of video frames. A performance improvement may be achieved, at least in part, by applying a temporal filter to an output of the multiclass classifier. For example, the temporal filter may leverage classifications associated with preceding input signals to improve the final classification given to a subsequent signal. In some embodiments, the temporal filter may also use data from a confusion matrix to correct for the probable occurrence of certain types of classification errors. The temporal filter may be a linear filter, a nonlinear filter, an adaptive filter, and/or a statistical filter.

Using temporal filters for automated real-time classification

Stitching of multiple images into a composite representation can be performed using a set of stitching parameters determined based, at least in part, upon a subjective stitching quality assessment value. A stitched image can be compared against its constituent images to obtain one or more objective quality metrics. These objective quality metrics can be fed, as input, to a trained classifier, which can infer a subjective quality assessment metric for the stitched (or otherwise composited) image. This subjective quality assessment metric can be used to adjust one or more compositing parameter values in order to provide at least a minimum subjective quality assessment value for composited images.

Stitching quality assessment for surround view systems

Systems and methods are disclosed herein for implementation of a vehicle command operation system that may use multi-modal technology to authenticate an occupant of the vehicle to authorize a command and receive natural language commands for vehicular operations. The system may utilize sensors to receive data indicative of a voice command from an occupant of the vehicle. The system may receive second sensor data to aid in the determination of the corresponding vehicular operation in response to the received command. The system may retrieve authentication data for the occupants of the vehicle. The system authenticates the occupant to authorize a vehicular operation command using a neural network based on at least one of the first sensor data, the second sensor data, and the authentication data. Responsive to the authentication, the system may authorize the operation to be performed in the vehicle based on the vehicular operation command.

Systems and methods for performing commands in a vehicle using speech and image recognition

In various examples, color statistic(s) from ground projections are used to harmonize color between reference and target frames representing an environment. The reference and target frames may be projected onto a representation of the ground (e.g., a ground plane) of the environment, an overlapping region between the projections may be identified, and the portion of each projection that lands in the overlapping region may be taken as a corresponding ground projection. Color statistics (e.g., mean, variance, standard deviation, kurtosis, skew, correlation(s) between color channels) may be computed from the ground projections (or a portion thereof, such as a majority cluster) and used to modify the colors of the target frame to have updated color statistics that match those from the ground projection of the reference frame, thereby harmonizing color across the reference and target frames.

Image stitching with color harmonization for surround view systems and applications

Systems and methods for determining the gaze direction of a subject and projecting this gaze direction onto specific regions of an arbitrary three-dimensional geometry. In an exemplary embodiment, gaze direction may be determined by a regression-based machine learning model. The determined gaze direction is then projected onto a three-dimensional map or set of surfaces that may represent any desired object or system. Maps may represent any three-dimensional layout or geometry, whether actual or virtual. Gaze vectors can thus be used to determine the object of gaze within any environment. Systems can also readily and efficiently adapt for use in different environments by retrieving a different set of surfaces or regions for each environment.

Neural network based determination of gaze direction using spatial models

In various examples, estimated field of view or gaze information of a user may be projected external to a vehicle and compared to vehicle perception information corresponding to an environment outside of the vehicle. As a result, interior monitoring of a driver or occupant of the vehicle may be used to determine whether the driver or occupant has processed or seen certain object types, environmental conditions, or other information exterior to the vehicle. For a more holistic understanding of the state of the user, attentiveness and/or cognitive load of the user may be monitored to determine whether one or more actions should be taken. As a result, notifications, AEB system activations, and/or other actions may be determined based on a more complete state of the user as determined based on cognitive load, attentiveness, and/or a comparison between external perception of the vehicle and estimated perception of the user.

Occupant attentiveness and cognitive load monitoring for autonomous and semi-autonomous driving applications

In various examples, audio alerts of emergency response vehicles may be detected and classified using audio captured by microphones of an autonomous or semi-autonomous machine in order to identify travel directions, locations, and/or types of emergency response vehicles in the environment. For example, a plurality of microphone arrays may be disposed on an autonomous or semi-autonomous machine and used to generate audio signals corresponding to sounds in the environment. These audio signals may be processed to determine a location and/or direction of travel of an emergency response vehicle (e.g., using triangulation). Additionally, to identify siren types—and thus emergency response vehicle types corresponding thereto—the audio signals may be used to generate representations of a frequency spectrum that may be processed using a deep neural network (DNN) that outputs probabilities of alert types being represented by the audio data.

Emergency response vehicle detection for autonomous driving applications

Apparatuses, systems, and techniques are described to determine locations of objects using images including digital representations of those objects. In at least one embodiment, a gaze of one or more occupants of a vehicle is determined independently of a location of one or more sensors used to detect those occupants.

Gaze detection using one or more neural networks

State information can be determined for a subject that is robust to different inputs or conditions. For drowsiness, facial landmarks can be determined from captured image data and used to determine a set of blink parameters. These parameters can be used, such as with a temporal network, to estimate a state (e.g., drowsiness) of the subject. To improve robustness, an eye state determination network can determine eye state from the image data, without reliance on intermediate landmarks, that can be used, such as with another temporal network, to estimate the state of the subject. A weighted combination of these values can be used to determine an overall state of the subject. To improve accuracy, individual behavior patterns and context information can be utilized to account for variations in the data due to subject variation or current context rather than changes in state.

Robust state estimation

Systems and methods for a self-adjusting vehicle mirror. The mirror automatically locates the face of the driver or another passenger, and orients the mirror to provide the driver/passenger face with a desired view from the mirror. The mirror may continue to reorient itself as the driver or passenger shifts position, to continuously provide a desired field of view even as he or she changes position over time. In certain embodiments, the mirror system of the disclosure can be a self-contained system, with the mirror, mirror actuator, camera, and computing device all contained within the mirror housing as a single integrated unit.

Automatically-adjusting mirror for use in vehicles

Interactions with virtual systems may be difficult when users inadvertently fail to provide sufficient information to proceed with their requests. Certain types of inputs, such as auditory inputs, may lack sufficient information to properly provide a response to the user. Additional information, such as image data, may enable user gestures or poses to supplement the auditory inputs to enable response generation without requesting additional information from users.

Multi-modal sensor fusion for content identification in applications of human-machine interfaces

In various examples, systems and methods are disclosed herein for a vehicle command operation system that may use technology across multiple modalities to cause vehicular operations to be performed in response to determining a focal point based on a gaze of an occupant. The system may utilize sensors to receive first data indicative of an eye gaze of an occupant of the vehicle. The system may utilize sensors to receive second data indicative of other data from the occupant. The system may then calculate a gaze vector based on the data indicative of the eye gaze of the occupant. The system may determine a focal point based on the gaze vector. In response to determining the focal point, the system causes an operation to be performed in the vehicle based on the second data.

Systems and methods for performing operations in a vehicle using gaze detection

In various examples, systems and methods are disclosed that accurately identify driver and passenger in-cabin activities that may indicate a biomechanical distraction that prevents a driver from being fully engaged in driving a vehicle. In particular, image data representative of an image of an occupant of a vehicle may be applied to one or more deep neural networks (DNNs). Using the DNNs, data indicative of key point locations corresponding to the occupant may be computed, a shape and/or a volume corresponding to the occupant may be reconstructed, a position and size of the occupant may be estimated, hand gesture activities may be classified, and/or body postures or poses may be classified. These determinations may be used to determine operations or settings for the vehicle to increase not only the safety of the occupants, but also of surrounding motorists, bicyclists, and pedestrians.

Growing community of inventors

Saratoga, CA, United States of America

Niranjan Avadhanam