Peking University

Wenmin Wang

Ronggang Wang

Wen Gao

Zhenyu Wang

Ge Li

Shengfu Dong

Ying Li

Hui Zhao

Tiejun Huang

Yang Zhao

Mengdi Fan

Siwei Ma

Yihao Zhang

Peilei Dong

Mingmin Zhen

Xianghao Zang

Ruonan Zhang

Lei Chen

Zhihao Li

Hongmeng Song

Liang Han

Jianbo Jiao

Chunbiao Zhu

Lu Ran

The present application discloses a cross-media retrieval method based on deep semantic space, which includes a feature generation stage and a semantic space learning stage. In the feature generation stage, a CNN visual feature vector and an LSTM language description vector of an image are generated by simulating a perception process of a person for the image; and topic information about a text is explored by using an LDA topic model, thus extracting an LDA text topic vector. In the semantic space learning phase, a training set image is trained to obtain a four-layer Multi-Sensory Fusion Deep Neural Network, and a training set text is trained to obtain a three-layer text semantic network, respectively. Finally, a test image and a text are respectively mapped to an isomorphic semantic space by using two networks, so as to realize cross-media retrieval. The disclosed method can significantly improve the performance of cross-media retrieval.

Cross-media retrieval method based on deep semantic space

A video action detection method based on a convolutional neural network (CNN) is disclosed in the field of computer vision recognition technologies. A temporal-spatial pyramid pooling layer is added to a network structure, which eliminates limitations on input by a network, speeds up training and detection, and improves performance of video action classification and time location. The disclosed convolutional neural network includes a convolutional layer, a common pooling layer, a temporal-spatial pyramid pooling layer and a full connection layer. The outputs of the convolutional neural network include a category classification output layer and a time localization calculation result output layer. The disclosed method does not require down-sampling to obtain video clips of different durations, but instead utilizes direct input of the whole video at once, improving efficiency. Moreover, the network is trained by using video clips of the same frequency without increasing differences within a category, thus reducing the learning burden of the network, achieving faster model convergence and better detection.

Video action detection method based on convolutional neural network

A method and a device for MCMC framework-based sub-hypergraph matching are provided. Matching of object features is performed by constructing sub-hypergraphs. In a large number of actual images and videos, objects vary constantly, and contain various noise points as well as other interference factors, which makes image object matching and searching very difficult. Perform object feature matching by representing the appearance and positions of objects by sub-hypergraphs allows for faster and more accurate image matching. Furthermore, a sub-hypergraph has several advantages over a graph or a hypergraph: on one hand, a sub-hypergraph has more geometric information (e.g. angle transformation, rotation, scale, etc.) than a graph, and has a lower degree of difficulty and better extensibility than a hypergraph. On the other hand, the disclosed method and device have stronger capabilities to resist interference and good robustness, and are adaptable to more complex settings, especially with outliers.

MCMC framework-based sub-hypergraph matching method and device

An image feature extraction method for person re-identification includes performing person re-identification by means of aligned local descriptor extraction and graded global feature extraction; performing the aligned local descriptor extraction by processing an original image by affine transformation and performing a summation pooling operation on image block features of same regions to obtain an aligned local descriptor; reserving spatial information between inner blocks of the image for the aligned local descriptor; and performing the graded global feature extraction by grading a positioned pedestrian region block and solving a corresponding feature mean value to obtain a global feature. The method can resolve the problem of feature misalignment caused by posture changes of pedestrian, etc., and eliminate the effect of unrelated backgrounds on re-recognition, thus improving the precision and robustness of person re-identification.

Image feature extraction method for person re-identification

A back-propagation significance detection method based on depth map mining, comprising: for an input image I, at a preprocessing phase, obtaining a depth image Iand an image Cwith four background corners removed of the image I; at a first processing phase, carrying out positioning detection on a significant region of the image by means of the obtained image Cwith four background corners removed and the obtained depth image Ito obtain the preliminary detection result Sof a significant object in the image; then carrying out depth mining on a plurality of processing phases of the depth image Ito obtain corresponding significance detection results; and then optimizing the significance detection result mined in each processing phase by means of a back-propagation mechanism to obtain a final significance detection result map. The method can improve the detection accuracy of the significance object.

Back-propagation image visual saliency detection method based on depth image mining

A bidirectional image-text retrieval method based on a multi-view joint embedding space includes: performing retrieval with reference to a semantic association relationship at a global level and a local level, obtaining the semantic association relationship at the global level and the local level in a frame-sentence view and a region-phrase view, and obtaining semantic association information in a global level subspace of frame and sentence in the frame-sentence view, obtaining semantic association information in a local level subspace of region and phrase in the region-phrase view, processing data by a dual-branch neural network in the two views to obtain an isomorphic feature and embedding the same in a common space, and using a constraint condition to reserve an original semantic relationship of the data during training, and merging the two semantic association relationships using multi-view merging and sorting to obtain a more accurate semantic similarity between data.

Method of bidirectional image-text retrieval based on multi-view joint embedding space

Disclosed is a deep discriminative network for person re-identification in an image or a video. Concatenation are carried out on different input images on a color channel by constructing a deep discriminative network, and an obtained splicing result is defined as an original difference space of different images. The original difference space is sent into a convolutional network. The network outputs the similarity between two input images by learning difference information in the original difference space, thereby realizing person re-identification. The features of an individual image are not learnt, and concatenation are carried out on input images on a color channel at the beginning, and difference information is learnt on an original space of the images by using a designed network. By introducing an Inception module and embedding the same into a model, the learning ability of a network can be improved, and a better differentiation effect can be achieved.

Method of using deep discriminate network model for person re-identification in image or video

The present disclosure provides a hybrid framework-based image bit-depth expansion method and device. The invention fuses a traditional de-banding algorithm and a depth network-based learning algorithm, and can remove unnatural effects in an image flat area whilst more realistically restoring numerical information of missing bits. The method comprises the extraction of image flat areas, local adaptive pixel value adjustment-based flat area bit-depth expansion and convolutional neural network-based non-flat area bit-depth expansion. The present invention uses a learning-based method to train an effective depth network to solve the problem of realistically restoring missing bits, whilst using a simple and robust local adaptive pixel value adjustment method in an flat area to effectively inhibit unnatural effects in the flat area such as banding, a ringing and flat noise, improving subjective visual quality of the flat area.

Hybrid framework-based image bit-depth expansion method and device

Disclosed is a method for detecting pedestrians in an image by using Gaussian penalty. Initial pedestrian boundary box is screened using a Gaussian penalty, to improve the pedestrian detection performance, especially sheltered pedestrians in an image. The method includes acquiring a training data set, a test data set and pedestrian labels of a pedestrian detection image; using the training data set for training to obtain a detection model by using a pedestrian detection method, and acquiring initial pedestrian boundary box and confidence degrees and coordinates thereof; performing Gaussian penalty on the confidence degrees of the pedestrian boundary box, to obtain confidence degree of the pedestrian boundary box after the penalty; and obtaining final pedestrian boundary boxes by screening the pedestrian boundary boxes. Thus, repeated boundary boxes of a single pedestrian are removed while reserving boundary boxes of sheltered pedestrians, thereby realizing the detection of the pedestrians in an image.

Method for detecting pedestrians in image by using Gaussian penalty

A Collaborative Deep Network model method for pedestrian detection includes constructing a new collaborative multi-model learning framework to complete a classification process during pedestrian detection; and using an artificial neuron network to integrate judgment results of sub-classifiers in a collaborative model, and training the network by means of the method for machine learning, so that information fed back by sub-classifiers can be more effectively synthesized. A re-sampling method based on a K-means clustering algorithm can enhance the classification effect of each classifier in the collaborative model, and thus improves the overall classification effect. By building a collaborative deep network model, different types of training data sets obtained using a clustering algorithm are used for training a plurality of deep network models in parallel, and then classification results, on deep network models, of an original data set are integrated and comprehensively analyzed, which achieves more accurate sample classification.

Collaborative deep network model method for pedestrian detection

A cross-media search method using a VGG convolutional neural network (VGG net) to extract image features. The 4096-dimensional feature of a seventh fully-connected layer (fc7) in the VGG net, after processing by a ReLU activation function, serves as image features. A Fisher Vector based on Word2vec is utilized to extract text features. Semantic matching is performed on heterogeneous images and the text features by means of logistic regression. A correlation between the two heterogeneous features, which are images and text, is found by means of semantic matching based on logistic regression, and thus cross-media search is achieved. The feature extraction method can effectively indicate deep semantics of image and text, improve cross-media search accuracy, and thus greatly improve the cross-media search effect.

Cross-media search method

A method for image matching includes acquiring a template image and a target image; acquiring a group of template features according to the template image; extracting a group of target features according to the target image; and according to template features and target features, calculating an degree of image similarity between the template image and each frame of target images, and using a target image with the maximum degree of image similarity as a matched image to the template image. In the image-matching method, image matching is performed by calculating an degree of image similarity between a template image and each target image according to a degree of image similarity between template features and target features, so that non-redundancy of features in an image matching process and correct image matching can be guaranteed, and the image matching accuracy can be improved.

Method and a device for image matching

A method and a device for post-processing depth/disparity maps adopt a strategy of combining edge information and segmentation information when detecting irregular edge regions. The method includes dividing a color image into super pixels when performing image segmentation on a color image; partitioning a grayscale range into a preset number of intervals, and for each super pixel, statistically obtaining a histogram of all the pixel points that fall within the intervals; determining, in a current super pixel, whether a ratio of the number of pixels contained in the interval having a maximum interval distribution value, to the total number of pixels in the current super pixel is less than the first threshold; and if so, further dividing the current super pixel using a color-based segmentation method. The disclosed method and device improve accuracy of color image division while ensuring image processing speed, thus improving detection accuracy of the irregular edge regions.

Depth/disparity map post-processing method and device

An image representation method and processing device based on local PCA whitening. A first mapping module maps words and characteristics to a high-dimension space. A principal component analysis module conducts principal component analysis in each corresponding word space, to obtain a projection matrix. A VLAD computation module computes a VLAD image representation vector; a second mapping module maps the VLAD image representation vector to the high-dimension space. A projection transformation module conducts projection transformation on the VLAD image representation vector obtained by means of projection. A normalization processing module conducts normalization on characteristics obtained by means of projection transformation, to obtain a final image representation vector. An obtained image representation vector is projected to a high-dimension space first, then projection transformation is conducted on a projection matrix computed in advance and vectors corresponding to words, to obtain a low-dimension vector; and in this way, the vectors corresponding to the words are consistent. The disclosed method and the processing device can obtain better robustness and higher performance.

Image representation method and processing device based on local PCA whitening

Disclosed in the present invention is a video foreground extraction method for a surveillance video, which adjusts a size of a block to adapt to different video resolutions based on an image block processing method; and then extracts a foreground object in a moving state by establishing a background block model, the method comprising: representing each frame of image I in the surveillance video as a block; initializing; updating a block background weight, a block temporary background and a temporary background; updating a block background and a background; saving a foreground, and updating a foreground block weight and a foreground block; and performing binarization processing on the foreground to obtain a final foreground result. The invention reduces the dimension of each frame of image in the surveillance video, eliminates the influence of the illumination on the block method by means of the pixel-based comparison, and meanwhile eliminates the influence on video analysis due to increasing size of the surveillance video. The invention can extract a dynamic foreground object, adapt to a larger size of a surveillance video, and has a lower algorithm complexity.

Growing community of inventors

Shenzhen, China

Wenmin Wang