Nvidia Corporation

Omnivision Technologies, Inc.

Beijing Qingzhouzhihang Intelligent Technology Co., Ltd

Xiaodong Yang

Jan Kautz

Ming-Yu Liu

Pavlo Molchanov

Guansong Liu

Chin Poh Pang

Xitong Yang

Shalini De Mello

Deqing Sun

Zhiding Yu

Da Meng

Fanyi Xiao

Jizhang Shan

Peng Lin

Kihwan Kim

Sifei Liu

Jinwei Gu

Boyang Zhang

Stephen Walter Tyree

Hongjun Li

Arash Vahdat

Yang Zou

Chenxu Luo

Masao Sambongi

Behrooz Mahasseni

Sergey Tulyakov

Tanmay Gupta

Chen Zhang

Zhongheng Ren

In an embodiment, an image sensor comprises a pixel array having a minimal repeating unit, where the minimal repeating unit consists of 4×4 pixels including 12 green pixels, 2 blue pixels, and 2 red pixels, where a minimal repeating unit is immediately next to another minimal repeating unit in row and column directions. In another embodiment, an image sensor comprises a pixel array having a minimal repeating unit, where the minimal repeating unit consists of 8×8 pixels including 48 green pixels, 8 blue pixels, and 8 red pixels.

Image sensor pixel array having minimal repeating unit

Apparatuses, systems, and techniques to detect object in images including digital representations of those objects. In at least one embodiment, one or more objects are detected in an image based, at least in part, on one or more pseudo-labels corresponding to said one or more objects.

Weakly-supervised object detection using one or more neural networks

The disclosure provides a framework or system for learning visual representation using a large set of image/text pairs. The disclosure provides, for example, a method of visual representation learning, a joint representation learning system, and an artificial intelligence (AI) system that employs one or more of the trained models from the method or system. The AI system can be used, for example, in autonomous or semi-autonomous vehicles. In one example, the method of visual representation learning includes: (1) receiving a set of image embeddings from an image representation model and a set of text embeddings from a text representation model, and (2) training, employing mutual information, a critic function by learning relationships between the set of image embeddings and the set of text embeddings.

Joint representation learning from images and text

A method and a device for self-supervised learning, a storage medium, and an electronic device are provided. The method includes: organizing real points in one column along a vertical direction into a pillar; determining a predicted point in a next frame; determining a first loss term based on a minimum distance among distances between predicted points in the next frame and real points in the next frame, and generating a loss function including the first loss term; and performing self-supervised learning processing based on the loss function. A pillar motion parameter representing motion of a real point is determined with the pillar as a unit, so as to enhance correlation between point clouds. Self-supervised learning can be realized in a case of no precise correspondence between the predicted point and the real point, and training is performed based on a large number of unlabeled point clouds.

Method and system for self-supervised learning of pillar motion for autonomous driving

A method, computer readable medium, and system are disclosed for visual sequence learning using neural networks. The method includes the steps of replacing a non-recurrent layer within a trained convolutional neural network model with a recurrent layer to produce a visual sequence learning neural network model and transforming feedforward weights for the non-recurrent layer into input-to-hidden weights of the recurrent layer to produce a transformed recurrent layer. The method also includes the steps of setting hidden-to-hidden weights of the recurrent layer to initial values and processing video image data by the visual sequence learning neural network model to generate classification or regression output data.

Transforming convolutional neural networks for visual sequence learning

Iterative prediction systems and methods for the task of action detection process an inputted sequence of video frames to generate an output of both action tubes and respective action labels, wherein the action tubes comprise a sequence of bounding boxes on each video frame. An iterative predictor processes large offsets between the bounding boxes and the ground-truth.

Iterative spatio-temporal action detection in video

There are numerous features in video that can be detected using computer-based systems, such as objects and/or motion. The detection of these features, and in particular the detection of motion, has many useful applications, such as action recognition, activity detection, object tracking, etc. The present disclosure provides a neural network that learns motion from unlabeled video frames. In particular, the neural network uses the unlabeled video frames to perform self-supervised hierarchical motion learning. The present disclosure also describes how the learned motion can be used in video action recognition.

Self-supervised hierarchical motion learning for video action recognition

Object re-identification refers to a process by which images that contain an object of interest are retrieved from a set of images captured using disparate cameras or in disparate environments. Object re-identification has many useful applications, particularly as it is applied to people (e.g. person tracking). Current re-identification processes rely on convolutional neural networks (CNNs) that learn re-identification for a particular object class from labeled training data specific to a certain domain (e.g. environment), but that do not apply well in other domains. The present disclosure provides cross-domain disentanglement of id-related and id-unrelated factors. In particular, the disentanglement is performed using a labeled image set and an unlabeled image set, respectively captured from different domains but for a same object class. The identification-related features may then be used to train a neural network to perform re-identification of objects in that object class from images captured from the second domain.

Cross-domain image processing for object re-identification

An image sensor pixel includes a plurality of photodiodes, a shared microlens, and a plurality of microlenses. The plurality of photodiodes are arranged as a photodiode array with each of the plurality of photodiodes disposed within a semiconductor material. The shared microlens is optically aligned with a group of neighboring photodiodes included in the plurality of photodiodes. Each of the plurality of microlenses are optically aligned with an individual one of the plurality of photodiodes other than the group of neighboring photodiodes. The plurality of microlenses laterally surrounds the shared microlens.

Image sensor with shared microlens between multiple subpixels

An image sensor includes a plurality of photodiodes, a plurality of color filters, and a plurality of microlenses. The plurality of photodiodes are arranged as a photodiode array, each of the plurality of photodiodes disposed within respective portions of a semiconductor material with a first lateral area. The plurality of color filters are arranged as a color filter array optically aligned with the photodiode array. Each of the plurality of color filters having a second lateral area greater than the first lateral area. The plurality of microlenses are arranged as a microlens array optically aligned with the color filter array and the photodiode array. Each of the plurality of microlenses have a third later area greater than the first lateral area and less than the second lateral area.

Image sensor with shared microlens

Detection of activity in video content, and more particularly detecting in video start and end frames inclusive of an activity and a classification for the activity, is fundamental for video analytics including categorizing, searching, indexing, segmentation, and retrieval of videos. Existing activity detection processes rely on a large set of features and classifiers that exhaustively run over every time step of a video at multiple temporal scales, or as a small improvement computationally propose segments of the video on which to perform classification. These existing activity detection processes, however, are computationally expensive, particularly when trying to achieve activity detection accuracy, and moreover are not configurable for any particular time or computation budget. The present disclosure provides a time and/or computation budget-aware method for detecting activity in video that relies on a recurrent neural network implementing a learned policy.

Budget-aware method for detecting activity in video

A method, computer readable medium, and system are disclosed for action video generation. The method includes the steps of generating, by a recurrent neural network, a sequence of motion vectors from a first set of random variables and receiving, by a generator neural network, the sequence of motion vectors and a content vector sample. The sequence of motion vectors and the content vector sample are sampled by the generator neural network to produce a video clip.

System and method for content and motion controlled action video generation

A method, computer readable medium, and system are disclosed for estimating optical flow between two images. A first pyramidal set of features is generated for a first image and a partial cost volume for a level of the first pyramidal set of features is computed, by a neural network, using features at the level of the first pyramidal set of features and warped features extracted from a second image, where the partial cost volume is computed across a limited range of pixels that is less than a full resolution of the first image, in pixels, at the level. The neural network processes the features and the partial cost volume to produce a refined optical flow estimate for the first image and the second image.

Growing community of inventors

San Jose, CA, United States of America

Xiaodong Yang