Nvidia Corporation

Yuzhuo Ren

Niranjan Avadhanam

Sakthivel Sivaraman

Hairong Jiang

Feng Hu

Sujay Yadawadkar

Siyue Wu

Dawid Stanislaw Pajak

Nuri Murat Arar

Rajath Shetty

Nishant Puri

Atousa Torabi

Shubhadeep Das

Jason Conrad Roche

Sumit Kumar Bhattacharya

Varsha Chandrashekhar Hedau

Nitin Bharadwaj

Chun-Wei Chen

Guangli Dai

Arjun Guru

Ankit Pashiney

In various examples, color harmonization is applied to images of an environment in a reference light space. For example, different cameras on an ego-object may use independent capturing algorithms to generate processed images of the environment representing a common time slice using different capture configuration parameters. The processed images may be transformed into deprocessed images by inverting one or more stages of image processing to transform the processed images into a reference light space of linear light, and color harmonization may be applied to the deprocessed images in the reference light space. After applying color harmonization, corresponding image processing may be reapplied to the harmonized images using corresponding capture configuration parameters, the resulting processed harmonized images may be stitched into a stitched image, and a visualization of the stitched image may be presented (e.g., on a monitor visible to an occupant or operator of the ego-object).

Image stitching with color harmonization of de-processed images for surround view systems and applications

In various examples, occupant assessment using multi-modal sensor fusion for monitoring systems and applications are provided. In some embodiments, an occupant monitoring system comprises an occupant evaluation function that may predict at least one characteristic representative of a size of the occupant. The occupant evaluation function may include a first processing path that generates a representation of features corresponding to the occupant based on optical image data, and a second processing path that performs operations to determine a depth corresponding to the one or more features based on depth data derived from the optical image data and the point cloud depth data. In some embodiments, a three-dimensional pose detection model generates a three-dimensional pose estimate of the occupant using the optical image data, and the three-dimensional pose estimate is scaled to an absolute pose based on the point cloud depth data.

Occupant evaluation using multi-modal sensor fusion for in-cabin monitoring systems and applications

In various examples, calibration techniques for interior depth sensors and image sensors for in-cabin monitoring systems and applications are provided. An intermediary coordinate system may be generated using calibration targets distributed within an interior space to reference 3D positions of features detected by both depth-perception and optical image sensors. Rotation-translation transforms may be determined to compute a first transform (H1) between the depth-perception sensor's 3D coordinate system and the 3D intermediary coordinate system, and a second transform (H2) between the optical image sensor's 2D coordinate system and the intermediary coordinate system. A third transform (H3) between the depth-perception sensor's 3D coordinate system and the optical image sensor's 2D coordinate system can be computed as a function of H1 and H2. The calibration targets may comprise a structural substrate that includes one or more fiducial point markers and one or more motion targets.

Multi-modal sensor calibration for in-cabin monitoring systems and applications

Stitching of multiple images into a composite representation can be performed using a set of stitching parameters determined based, at least in part, upon a subjective stitching quality assessment value. A stitched image can be compared against its constituent images to obtain one or more objective quality metrics. These objective quality metrics can be fed, as input, to a trained classifier, which can infer a subjective quality assessment metric for the stitched (or otherwise composited) image. This subjective quality assessment metric can be used to adjust one or more compositing parameter values in order to provide at least a minimum subjective quality assessment value for composited images.

Stitching quality assessment for surround view systems

In various examples, color statistic(s) from ground projections are used to harmonize color between reference and target frames representing an environment. The reference and target frames may be projected onto a representation of the ground (e.g., a ground plane) of the environment, an overlapping region between the projections may be identified, and the portion of each projection that lands in the overlapping region may be taken as a corresponding ground projection. Color statistics (e.g., mean, variance, standard deviation, kurtosis, skew, correlation(s) between color channels) may be computed from the ground projections (or a portion thereof, such as a majority cluster) and used to modify the colors of the target frame to have updated color statistics that match those from the ground projection of the reference frame, thereby harmonizing color across the reference and target frames.

Image stitching with color harmonization for surround view systems and applications

In various examples, estimated field of view or gaze information of a user may be projected external to a vehicle and compared to vehicle perception information corresponding to an environment outside of the vehicle. As a result, interior monitoring of a driver or occupant of the vehicle may be used to determine whether the driver or occupant has processed or seen certain object types, environmental conditions, or other information exterior to the vehicle. For a more holistic understanding of the state of the user, attentiveness and/or cognitive load of the user may be monitored to determine whether one or more actions should be taken. As a result, notifications, AEB system activations, and/or other actions may be determined based on a more complete state of the user as determined based on cognitive load, attentiveness, and/or a comparison between external perception of the vehicle and estimated perception of the user.

Occupant attentiveness and cognitive load monitoring for autonomous and semi-autonomous driving applications

Apparatuses, systems, and techniques are described to determine locations of objects using images including digital representations of those objects. In at least one embodiment, a gaze of one or more occupants of a vehicle is determined independently of a location of one or more sensors used to detect those occupants.

Gaze detection using one or more neural networks

State information can be determined for a subject that is robust to different inputs or conditions. For drowsiness, facial landmarks can be determined from captured image data and used to determine a set of blink parameters. These parameters can be used, such as with a temporal network, to estimate a state (e.g., drowsiness) of the subject. To improve robustness, an eye state determination network can determine eye state from the image data, without reliance on intermediate landmarks, that can be used, such as with another temporal network, to estimate the state of the subject. A weighted combination of these values can be used to determine an overall state of the subject. To improve accuracy, individual behavior patterns and context information can be utilized to account for variations in the data due to subject variation or current context rather than changes in state.

Robust state estimation

Interactions with virtual systems may be difficult when users inadvertently fail to provide sufficient information to proceed with their requests. Certain types of inputs, such as auditory inputs, may lack sufficient information to properly provide a response to the user. Additional information, such as image data, may enable user gestures or poses to supplement the auditory inputs to enable response generation without requesting additional information from users.

Multi-modal sensor fusion for content identification in applications of human-machine interfaces

Context-based state estimation

A neural network system leverages dual attention, specifically both spatial attention and channel attention, to jointly estimate heart rate and respiratory rate of a subject by processing images of the subject. A motion neural network receives images of the subject and estimates heart and breath rates of the subject using both spatial and channel domain attention masks to focus processing on particular feature data. An appearance neural network computes a spatial attention mask from the images of the subject and may indicate that features associated with the subject's face (as opposed to the subject's hair or shoulders) to accurately estimate the heart and/or breath rate. Channel-wise domain attention is learned during training and recalibrates channel-wise feature responses to select the most informative features for processing. The channel attention mask is learned during training and can be used for different subjects during deployment.

Joint estimation of heart rate and respiratory rate using neural networks

In various examples, two or more cameras in an automotive surround view system generate two or more input images to be stitched, or combined, into a single stitched image. In an embodiment, to improve the quality of a stitched image, a feedback module calculates two or more scores representing errors between the stitched image and one or more input images. If a computed score indicates structural errors in the stitched image, the feedback module calculates and applies one or more geometric transforms to apply to the one or more input images. If a computed score indicates color errors in the stitched image, the feedback module calculates and applies one or more photometric transforms to apply to the one or more input images.

Growing community of inventors

San Jose, CA, United States of America

Yuzhuo Ren