Altumview Systems Inc.

Microsoft Technology Licensing, LLC

Jie Liang

Chih-Lung Lin

Xing Xing Wang

Sridhar Srinivasan

Minghua Chen

Shankar Lakshmi Regunathan

Him Wai Ng

Thomas W Holcomb

Yu Gao

Mehdi Seyfi

Rui Ma

Jianbing Wu

Chao Shen

Pohsiang Hsu

Minghui Xia

Jiannan Zheng

Ye Lu

Ming-Chieh Lee

Regis J Crinon

Ce Wang

Timothy E Onders

Weiming Wang

Andrew Tsun-Hong Au

Chi Chung Chan

Dong Zhang

Eric Honsch

Kaichao Wang

Zili Yi

Sami Ma

Shuo-Jen Wu

Zachary DeVries

The present disclosure provides a method and system for efficiently and securely managing human motion data. In one aspect, a human-motion-data-managing system first receives a sequence of video images including a first person. Next, for each image in the sequence of video images, the human-motion-data-managing system detects the first person in the video image; and subsequently extracts a skeleton figure of the detected first person from the image, wherein the skeleton figure is composed of a set of human keypoints. Next, human-motion-data-managing system combines a sequence of extracted skeleton figures of the detected first person from the sequence of video images to form a skeleton sequence of the detected first person which depicts a continuous motion of the first person. The human-motion-data-managing system subsequently transmits the skeleton sequence to a server in place of the actual images of the first person to preserve the privacy of the first person.

Privacy-preserving human action recognition, storage, and retrieval via joint edge and cloud computing

Embodiments described herein provide various examples of real-time visual object tracking. In another aspect, a process for performing a local re-identification of a target object which was earlier detected in a video but later lost when tracking the target object is disclosed. This process begins by receiving a current video frame of the video and a predicted location of the target object. The process then places a current search window in the current video frame centered on or in the vicinity of the predicted location of the target object. Next, the process extracts a feature map from an image patch within the current search window. The process further retrieves a set of stored feature maps computed at a set of previously-determined locations of the target object from a set of previously-processed video frames in the video. The process next computes a set of correlation maps between the feature map and each of the set of stored feature maps. The process then attempts to re-identify the target object locally in the current video frame based on the set of computed correlation maps.

Real-time visual object tracking for unmanned aerial vehicles (UAVs)

Embodiments described herein provide examples of a real-time visual object tracking system. In one aspect, an unmanned aerial vehicle (UAV) capable of performing real-time visual tracking of a moving object includes: a processor; a memory coupled to the processor; and a camera to capture a video of the moving object. This UAV additionally includes a visual tracking module to: receive a first video image and a first location of the object; receive a second video image following the first video image; place a first search window in the first video image and a second search window in the second video image centered on a second location in the second video image having the same coordinates as the first location; compute a correlation between an image patch within the first search window and an image patch within the second search window; and determine an updated location of the object in the second video image.

High-performance visual object tracking for embedded vision systems

Embodiments described herein provide various examples of a face-image training data preparation system for performing large-scale face-image training data acquisition, preprocessing, cleaning, balancing, and post-processing. The disclosed training data preparation system can collect a very large set of loosely-labeled images of different people from the public domain, and then generate a raw training dataset including a set of incorrectly-labeled face images. The disclosed training data preparation system can then perform cleaning and balancing operations on the raw training dataset to generate a high-quality face-image training dataset free of the incorrectly-labeled face images. The processed high-quality face-image training dataset can be subsequently used to train deep-neural-network-based face recognition systems to achieve high performance in various face recognition applications. Compared to conventional face recognition systems and techniques, the disclosed training data preparation system and technique provide a fully-automatic, highly-deterministic and high-quality training data preparation procedure which does not rely heavily on assumptions.

High-quality training data preparation for high-performance face recognition systems

Embodiments described herein provide various examples of an automatic obstacle avoidance system for unmanned vehicles using embedded stereo vision techniques. In one aspect, an unmanned aerial vehicle (UAV) capable of performing autonomous obstacle detection and avoidance is disclosed. This UAV includes: a stereo vision camera set coupled to the one or more processors and the memory to capture a sequence of stereo images; and a stereo vision module configured to: receive a pair of stereo images captured by a pair of stereo vision cameras; perform a border cropping operation on the pair of stereo images to obtain a pair of cropped stereo images; perform a sub sampling operation on the pair of cropped stereo images to obtain a pair of sub sampled stereo images; and perform a dense stereo matching operation on the pair of sub sampled stereo images to generate a dense three-dimensional (3D) point map of a space corresponding to the pair of stereo images.

Obstacle avoidance system based on embedded stereo vision for unmanned aerial vehicles

Embodiments described herein provide various examples of an age and gender estimation system capable of performing age and gender classifications on face images having sizes greater than the maximum number of input pixels supported by a given small-scale hardware convolutional neural network (CNN) module. In some embodiments, the proposed age and gender estimation system can first divide a high-resolution input face image into a set of image patches with judiciously designed overlaps among neighbouring patches. Each of the image patches can then be processed with a small-scale CNN module, such as the built-in CNN module in Hi3519 SoC. The outputs corresponding to the set of image patches can be subsequently merged to obtain the output corresponding to the input face image, and the merged output can be further processed by subsequent layers in the age and gender estimation system to generate age and gender classifications for the input face image.

Age and gender estimation using small-scale convolutional neural network (CNN) modules for embedded systems

Embodiments described herein provide various examples of a real-time face-detection, face-tracking, and face-pose-selection subsystem within an embedded video system. In one aspect, a process for performing real-time face-pose-estimation and best-pose selection for a detected person captured in a video is disclosed. This process includes the steps of: receiving a video image among a sequence of video frames of a video; performing a face detection operation on the video image to detect a set of faces in the video image; detecting a new person appears in the video based on the set of detected faces; tracking the new person through subsequent video images in the video by detecting a sequence of face images of the new person in the subsequent video images; and for each of the subsequent video images which contains a detected face of the new person being tracked: estimating a pose associated with the detected face and updating a best pose for the new person based on the estimated pose. Upon detecting that the new person has disappeared from the video, the process then transmits a detected face of the new person corresponding to the current best pose to a server, wherein transmitting the detected face having the best pose among the sequence of detected face images reduces network bandwidth and improves storage efficiency.

Method and apparatus for real-time face-tracking and face-pose-selection on embedded vision systems

Embodiments described herein provide various examples of a joint face-detection and head-pose-angle-estimation system based on using a small-scale hardware CNN module such as the built-in CNN module in HiSilicon Hi3519 system-on-chip. In some embodiments, the disclosed joint face-detection and head-pose-angle-estimation system is configured to jointly perform multiple tasks of detecting most or all faces in a sequence of video frames, generating pose-angle estimations for the detected faces, tracking detected faces of a same person across the sequence of video frames, and generating 'best-pose' estimation for the person being tracked. The disclosed joint face-detection and pose-angle-estimation system can be implemented on resource-limited embedded systems such as smart camera systems that are only integrated with one or more small-scale CNN modules. The proposed system in conjunction with using a subimage-based technique has made it possible to performance multiple face detection and face recognition tasks on high-resolution input images with small-scale low-cost CNN modules.

Joint face-detection and head-pose-angle-estimation using small-scale convolutional neural network (CNN) modules for embedded systems

Embodiments of a convolutional neural network (CNN) system based on using resolution-limited small-scale CNN modules are disclosed. In some embodiments, a CNN system includes: a receiving module for receiving an input image of a first image size, the receiving module can be used to partition the input image into a set of subimages of a second image size; a first processing stage that includes a first hardware CNN module configured with a maximum input image size, the first hardware CNN module is configured to sequentially receive the set of subimages and sequentially process the received subimages to generate a set of outputs; a merging module for merging the sets of outputs into a set of merged feature maps; and a second processing stage for receiving the set of feature maps and processing the set of feature maps to generate an output including at least one prediction on the input image.

Convolutional neural network (CNN) system based on resolution-limited small-scale CNN modules

Embodiments described herein provide various examples of a face detection system, based on using a small-scale hardware convolutional neural network (CNN) module configured into a multi-task cascaded CNN. In some embodiments, a subimage-based CNN system can be configured to be equivalent to a large-scale CNN that processes the entire input image without partitioning such that the output of the subimage-based CNN system can be exactly identical to the output of the large-scale CNN. Based on this observation, some embodiments of this patent disclosure make use of the subimage-based CNN system and technique on one or more stages of a cascaded CNN or a multitask cascaded CNN (MTCNN) so that a larger input image to a given stage of the cascaded CNN or the MTCNN can be partitioned into a set of subimages of a smaller size. As a result, each stage of the cascaded CNN or the MTCNN can use the same small-scale hardware CNN module that is associated with a maximum input image size constraint.

Face detection using small-scale convolutional neural network (CNN) modules for embedded systems

Techniques and tools are described for flexible range reduction of samples of video. For example, an encoder signals a first set of one or more syntax elements for range reduction of luma samples and signals a second set of one or more syntax elements for range reduction of chroma samples. The encoder selectively scales down the luma samples and chroma samples in a manner consistent with the first syntax element(s) and second syntax element(s), respectively. Or, an encoder signals range reduction syntax element(s) in an entry point header for an entry point segment, where the syntax element(s) apply to pictures in the entry point segment. If range reduction is used for the pictures, the encoder scales down samples of the pictures. Otherwise, the encoder skips the scaling down. A decoder performs corresponding parsing and scaling up operations.

Flexible range reduction

A video codec efficiently signals that a frame is identical to its reference frame, such that separate coding of its picture content is skipped. Information that a frame is skipped is represented jointly in a coding table of a frame coding type element for bit rate efficiency in signaling. Further, the video codec signals the picture type (e.g., progressive or interlaced) of skipped frames, which permits different repeat padding methods to be applied according to the picture type.

Signaling and repeat padding for skip frames

A decoder receives an entry point header comprising plural control parameters for an entry point segment corresponding to the entry point header. The entry point header is in an entry point layer of a bitstream comprising plural layers. The decoder decodes the entry point header. The plural control parameters can include various combinations of control parameters such as a pan scan on/off parameter, a reference frame distance on/off parameter, a loop filtering on/off parameter, a fast chroma motion compensation on/off parameter, an extended range motion vector on/off parameter, a variable sized transform on/off parameter, an overlapped transform on/off parameter, a quantization decision parameter, and an extended differential motion vector coding on/off parameter, a broken link parameter, a closed entry parameter, one or more coded picture size parameters, one or more range mapping parameters, a hypothetical reference decoder buffer parameter, and/or other parameter(s).

Signaling coding and display options in entry point headers

Tools and techniques for applying scan patterns during encoding and decoding of progressive video are described. For example, a video decoder entropy decodes transform coefficients in a one-dimensional array and scans the transform coefficients into a block according to a scan pattern. The block is 8&#xd7;4, and the scan pattern biases the vertical direction for at least the lowest frequency AC coefficients in the horizontal and vertical directions. Or, the block is 4&#xd7;8, and the scan pattern biases the horizontal direction for at least the lowest frequency AC coefficients in the horizontal and vertical directions. A corresponding video encoder applies the scan patterns to scan transform coefficients from blocks to one-dimensional arrays.

Growing community of inventors

Coquitlam, Canada

Jie Liang