Tsinghua University

Fujitsu Corporation

Guangdong Oppo Mobile Telecommunications Corp., Ltd.

Lu Fang

Qionghai Dai

Jinzhi Zhang

Tiankuang Zhou

Xiaoyun Yuan

Tao Yu

Mengqi Ji

Dawei Zhong

Leyao Liu

Tian Zheng

Yong Wang

Jun Sun

Ping Liu

Wei Cheng

Guangyu Wang

Lan Xu

Yebin Liu

Zhongguang Zheng

Siyuan Gu

Yingju Xia

Yinheng Zhu

Ruofan Tang

Shi Mao

Haiyang Ying

Zhihao Xu

Zequn Chen

Dingjian Jin

Haozhe Lin

Anke Zhang

In a method for generating a multi-dimensional scene graph with a complex light field, entity features of entities are obtained by inputting respective 2-Dimensional (2D) images captured in multiple view directions into an object detection model, and features of a respective single-view-direction scene graph are obtained by predicting a semantic relation among the entities contained in a corresponding 2D image captured in each view direction. An entity correlation for an entity among the multiple view directions is determined as an entity re-identification result. A multi-dimensional bounding box for the entity is established based on the entity re-identification result and a geometric constraint of camera parameters. A feature fusion result is obtained by fusing features of respective single-view-direction scene graphs in the multiple view directions. The multi-dimensional scene graph with the complex light field is established.

Method, system and apparatus for generating multi-dimensional scene graph with complex light field

The light field reconstruction method includes: obtaining a human segmentation result via a pre-trained semantic segmentation network, and obtaining an object segmentation result according to a pre-obtained scene background; fusing multiple frames of depth maps to obtain a geometric model, obtaining a complete human model according to a pre-trained human model completion network, and registering the models by point cloud registration and fusing the registered models to obtain an object model, so as to obtain a complete human model with geometric details and the object model; tracking motion of a rigid object through point cloud registration; reconstructing the complete human model with geometric details through a human skeleton tracking and a non rigid tracking of human surface nodes; and performing a fusion operation in time sequence and obtaining a reconstructed human model and a reconstructed rigid object model through the fusion operation.

Light field reconstruction method and apparatus of a dynamic scene

A system for variable illumination intelligent imaging of a billion pixel light field includes a support, a light source, an acquisition array and a control system. The light source and the acquisition array are mounted on the support. The acquisition array includes whole body acquisition cameras configured to acquire whole images of a human body and detail acquisition cameras configured to acquire area detail images of the human body at a plurality of angles of view. The control system, respectively connected to the light source and the acquisition array, is configured to synchronously control the acquisition array, and control the light source and the acquisition array to synchronously acquire a current human body whole image and a current human body detail image of an acquired person after each illumination environment conversion based on a preset light source illumination mode and a conversion frequency.

System and method for variable illumination intelligent imaging of billion pixel light field

A method includes: acquiring a semantic primitive set of a multi-view image set; acquiring a coordinate offset by inputting coordinate information and a feature vector corresponding to a first grid sampling point of the semantic primitive set into a first network model, and acquiring a second grid of the semantic primitive set based on the coordinate offset and geometric attribute information of the semantic primitive set; acquiring first feature information of a second grid sampling point by inputting coordinate information and a feature vector corresponding to the second grid sampling point, and an observation angle value into a second network model, and acquiring second feature information of the semantic primitive set based on the first feature information; and acquiring a light field reconstruction result of the multi-view image set based on an observation angle value of the semantic primitive set and third feature information extracted from the second feature information.

Method and system for semantic-driven intelligent reconstruction of large-scene sparse light field

A method for processing three-dimensional (3D) point cloud data based on incremental sparse 3D convolution is provided. A computer device obtains 3D point cloud data and forms a training set by processing the 3D point cloud data. The computer device constructs and trains a sparse 3D convolutional network model by inputting the training set. The computer device constructs an incremental sparse 3D convolutional network model by performing incremental replacement of sparse convolutional layers of the trained sparse 3D convolutional network model. The computer device inputs real-time 3D point cloud data into the incremental sparse 3D convolutional network model, and determines an output result as a result of processing the real-time 3D point cloud data. Processing of the 3D point cloud data at least includes 3D semantic segmentation, target detection, 3D classification and video processing.

Three-dimensional point cloud data processing method and device based on incremental sparse three-dimensional convolution

An intelligent understanding apparatus for real-time reconstruction of a large-scale scene light field includes the following. A data obtaining module obtains a 3D instance depth map, and obtain 3D voxels and voxel color information through simultaneous positioning and map generation. The model constructing module constructs and trains a real-time light field reconstruction network model using a ScanNet dataset. The real-time light field reconstruction network model extracts features of the 3D voxels and voxel color information, and obtain a semantic segmentation result and an instance segmentation result. The semantic segmentation module inputs the 3D voxel and voxel color information corresponding to the 3D instance depth map into the trained real-time light field reconstruction network model, and determine an output as a semantic segmentation result and an instance segmentation result corresponding to the 3D instance depth map.

Intelligent understanding system for real-time reconstruction of large-scale scene light field

A system for intelligent sensing and computing of an immersive light field includes a mobile platform, a support, an acquisition array, a control system and a computing system. The control system and the acquisition array are fixed on the support that is in form of hemisphere. The support changes a pose through the mobile platform. The acquisition array includes acquisition cameras of multiple views that are uniformly distributed and fixed on the support, and lenses of the acquisition cameras point to outside from a sphere center and are configured to acquire video images at a plurality of pitch angles. The control system is connected to the acquisition array and configured to transmit the video images to a control device for data storage, and the computing system is configured to acquire the video images in the control device and generate an immersive light field video based on the video images.

System and method for intelligent sensing and computing of immersive light field

Provided are a three-dimensional reconstruction method, apparatus and system of a dynamic scene, a server and a medium. The method includes: acquiring multiple continuous depth image sequences of the dynamic scene, where the multiple continuous depth image sequences are captured by an array of drones equipped with depth cameras; fusing the multiple continuous depth image sequences to establish a three-dimensional reconstruction model of the dynamic scene; obtaining target observation points of the array of drones through calculation according to the three-dimensional reconstruction model and current poses of the array of drones; and instructing the array of drones to move to the target observation points to capture, and updating the three-dimensional reconstruction model according to multiple continuous depth image sequences captured by the array of drones at the target observation points.

Dynamic scene three-dimensional reconstruction method, apparatus and system, server, and medium

A method and a device for gigapixel-level light field intelligent reconstruction of a large-scale scene are provided. The method includes: obtaining a coarse three-dimensional geometric model based on a multi-view three-dimensional reconstruction system; constructing an implicit representation of the meta-deformed manifold on the coarse three-dimensional geometric model; and optimizing the implicit representation of the meta-deformed manifold to obtain the light field reconstruction in the form of free viewpoint rendering of the large-scale scene.

Method and device for gigapixel-level light field intelligent reconstruction of large-scale scene

The present disclosure provides a material identification method and a device based on laser speckle and modal fusion, an electronic device and a non-transitory computer readable storage medium. The method includes: performing data acquisition on an object by using a structured light camera to obtain a color modal image, a depth modal image and an infrared modal image; preprocessing the color modal image, the depth modal image and the infrared modal image; and inputting the color modal image, the depth modal image and the infrared modal image preprocessed into a preset depth neural network for training, to learn a material characteristic from a speckle structure and a coupling relation between color modal and depth modal, to generate a material classification model for classifying materials, and to generate a material prediction result in testing by the material classification model of the object.

Material identification method and device based on laser speckle and modal fusion

A method for large scene elastic semantic representation and self-supervised light field reconstruction is provided. The method includes acquiring a first depth map set corresponding to a target scene, in which the first depth map set includes a first depth map corresponding to at least one 5 angle of view; inputting the first depth map set into a target elastic semantic reconstruction model to obtain a second depth map set, in which the second depth map set includes a second depth map corresponding to the at least one angle of view; and fusing the second depth map corresponding to the at least one angle of view to obtain a target scene point cloud corresponding to the target scene.

Method for large scene elastic semantic representation and self-supervised light field reconstruction

The present disclosure provides a multi-image-based image enhancement method and device, an electronic device and a non-transitory computer readable storage medium. The method includes: aligning a low-resolution target image and a reference image in an image domain; performing, an alignment in a feature domain; and synthesizing features corresponding to the low-resolution target image and features corresponding to the reference image to generate a final output.

Multi-image-based image enhancement method and device

The present disclosure relates to a method and device for generating a finite state automata for recognizing a chemical name in a text, and a recognition method. According to an embodiment of the present disclosure, the method comprises substituting representation constants of categories of character segments appearing in an organic compound name set into the organic compound name set to obtain a conversion name set; updating the conversion name set based on a conversion name segment which repeatedly appears in the conversion name set; and generating the finite state automata based on the updated conversion name set.

Generating finite state automata for recognition of organic compound names in Chinese

The disclosure provides 3D reconstruction methods and devices. The method includes: obtaining data captured by the camera and data captured by the inertial measurement unit; obtaining a pose of the camera based on the data; obtaining an adjustment value of the pose of the camera and an adjustment value of bias of the inertial measurement unit; updating the pose of the camera based on the adjustment value of the pose of the camera; determining whether the adjustment value of bias of the inertial measurement unit is less than a preset value; in response to the adjustment value of bias of the inertial measurement unit being greater than or equal to the preset value, determining that a current loop for 3-dimensional reconstruction is an error loop; removing the error loop; and constructing a 3-dimensional model for surroundings of the camera based on the updated pose of the camera and remaining loops.

3-dimensional reconstruction method, 3-dimensional reconstruction device, and storage medium

A method and an apparatus for camera-free light field video processing with all-optical neural network are disclosed. The method includes: mapping the light field video by a digital micro-mirror device (DMD) and an optical fiber coupler, a two-dimensional 2D spatial optical signal into a one-dimensional 1D input optical signal; realizing a multiply-accumulate computing model in a structure of all-optical recurrent neural network structure, and processing the 1D input signal to obtain the processed signal; and receiving the processed signal and outputting an electronic signal by a photodetector, or receiving the processed signal by a relay optical fiber for relay transmission of the processed signal. The method and system here realize light field video processing without the use of a camera and the whole system is all-optical, thus possessing the advantage in computing speed and energy-efficiency.

Growing community of inventors

Beijing, China

Lu Fang