Zhejiang University

Zhejiang Lab

Samsung Electronics Co., Ltd.

Huawei Technologies Co., Limited

Zhejiang Sensetime Technology Development Co., Ltd

Image Derivative Inc.

Hujun Bao

Hongsheng Wang

Guang Chen

Guofeng Zhang

Rui Wang

Wei Hua

Haomin Liu

Ho Young Lee

Jiwon Kim

Du Sik Park

Weiwei Xu

Xiaowei Zhou

Weiqiang Jia

Chao Ma

Jiong Zhou

Chen Li

Yong Li

Fei Yang

Jin Huang

Yazhen Yuan

Xiaogang Xu

Hongcai Cheng

Xiaolong Wang

Zhiyuan Yu

Lingfang Zeng

Bowen Tan

Jiaming Sun

Yunjin Zhang

Huanbo Zheng

Junting Dong

Qing Shuai

Jian Zhu

Xianzhong Fang

Kangkan Wang

Wenzhuo Yang

Yunsheng Luo

Hongcheng Zhao

Zihan Zhu

Qinhong Jiang

Jiamin Xu

Xiuchao Wu

Weiju Lan

Dejin He

Shi Li

Jianping Shi

Yuchi Huo

Chuankun Zheng

Qing Liao

Tianlei Hu

Shuibing He

Bowen Yu

The present disclosure provides a pipelining and parallelizing graph execution method for neural network model computation and apparatus, and provides a pipelining and parallelizing graph execution method for neural network model computation and apparatus in a deep learning training system. The method includes the graph execution flow in a neural network model computation process and a process of cooperative work of all functional modules. The pipelining and parallelizing graph execution method for neural network model computation includes creating a graph executive on a native machine according to a physical computation graph compiled and generated by a deep learning framework.

Pipelining and parallelizing graph execution method for neural network model computation and apparatus thereof

Disclosed in the present invention is an indoor scene virtual roaming method based on reflection decomposition, the method includes: firstly, by means of three-dimensional reconstruction, obtaining a rough global triangular mesh model projection as an initial depth map, aligning depth edges to color edges, and converting the aligned depth map into a simplified triangular mesh; checking planes in the global triangular mesh model, and if a certain plane is a reflection plane, constructing a double-layer expression in a reflection area for each picture in which the reflection plane is visible, so as to correctly render the reflection effect on an object surface; and giving a virtual viewport, using neighborhood pictures and the triangular mesh to draw a picture of the virtual viewport, and for the reflection area, using foreground and background pictures and foreground and background triangular meshes to perform drawing.

Indoor scene virtual roaming method based on reflection decomposition

Disclosed is a human motion capture method based on unsynchronized videos, which can effectively recover the 3D motion of a target person through multiple unsynchronized videos of the person. In order to utilize multiple unsynchronized videos, the present disclosure provides a video synchronization and motion reconstruction method. The present disclosure is implemented in the following steps: synchronizing multiple videos based on a 3D human pose; performing motion reconstruction based on synchronized videos, modeling the motion difference across different viewpoints by using the low-rank constraint to realize high-precision human motion capture from the plurality of unsynchronized videos. According to the present disclosure, more accurate motion capture is carried out by using multiple unsynchronized videos.

Motion capture method based on unsynchronized videos

Disclosed are a method and apparatus for constructing a three-dimensional data set of a pedestrian re-identification based on a neural radiation field. The method includes the following steps: S1: capturing images of pedestrians to be entered by a group of cameras at different viewing angles; S2: generating a three-dimensional spatial position point set by sampling through camera rays in the scenario, and converting observation directions of the cameras corresponding to the three-dimensional spatial position point set into three-dimensional Cartesian unit vectors; and S3: inputting, into a multi-layer sensor, the three-dimensional spatial position point set and the observation directions converted into the three-dimensional Cartesian unit vectors, to output corresponding densities and colors. The method and apparatus of the present disclosure gives a brand-new method for constructing a pedestrian re-identification data set, and provides a new idea of data set construction.

Method and apparatus for constructing three-dimensional data set of pedestrian re-identification based on neural radiation field

Provided is a video-based method and system for accurately estimating heart rate and facial blood volume distribution, and the method mainly comprises the following steps: firstly, carrying out face detection of video frame containing human face, and extracting face image sequence and face key position points sequence in time dimension; secondly, compressing these sequence of face image and face key position points to obtain the facial signals in time dimension; thirdly, estimating facial blood volume distribution by facial signals mentioned in third step; finally, estimating heart rate values by using model based on deep learning technology and the spectrum analysis method respectively, then fusing the estimation results by Kalman filter to promote the accuracy of heart rate estimation.

Video-based method and system for accurately estimating human body heart rate and facial blood volume distribution

The present invention discloses a dynamic rendering method and device based on implicit light transfer function, comprising the following steps: step 1, inserting an object into an original three-dimensional scene to form a new three-dimensional scene, using a new three-dimensional scene, a camera position, and an observation direction as input samples, and using the difference between the true value of the rendering result of the original three-dimensional scene and the true value of the rendering result of the new three-dimensional scene as a first sample label to form a first class of sample data; supervised learning of a neural network by using the first class of sample data, using a neural network with optimizable parameters as the implicit light transfer function; step 2, taking the new three-dimensional scene, the camera position, and the observation direction as input variables of the implicit light transfer function, obtaining a radiance change field of the new three-dimensional scene through the calculating of the implicit light transfer function, and merging the radiance change field into the rendering result of the original three-dimensional scene to obtain the rendering result of the new three-dimensional scene, suitable for dynamic rendering of various three-dimensional scenes.

Dynamic rendering method and device based on implicit light transfer function

A shader auto-simplifying method and system include: obtaining a rendering instruction flow, extracting a target shader from the rendering instruction flow, and creating a simplifying shader differing from the target shader in code only; intercepting a current frame of a rendering instruction with a rendering initiating instruction of the target shader as a particular frame; obtaining time consumed by the simplifying shader by measuring time needed for rendering the particular frame with the simplifying shader; obtaining error(s) of the simplifying shader by measuring a pixel difference value between a rendering frame drawn by the simplifying shader and the particular frame when a rendering instruction corresponding to the particular frame is executed; and screening an optimal simplifying shader according to the time consumed by the simplifying shader and the error of the simplifying shader.

Shader auto-simplifying method and system based on rendering instruction flow

A data acquisition and reconstruction method and a data acquisition and reconstruction system for human body three-dimensional modeling based on a single mobile phone. In the aspect of data acquisition, the present application only uses a single smart phone, and uses augmented reality technology to guide users to collect high-quality video data input for a reconstruction algorithm, so as to ensure that the subsequent human body reconstruction algorithm can stably obtain a high-quality three-dimensional human body model. In the aspect of a reconstruction algorithm, the present application designs a deformable implicit neural radiance field. The use of an implicit spatial deformation field estimation model solves the problem that the subject has small motion in the process of collecting data with a single mobile phone; the implicit signed distance field is used to represent human geometry, which has rich expressive ability and improves the accuracy of three-dimensional human model reconstruction.

Data acquisition and reconstruction method and system for human body three-dimensional modeling based on single mobile phone

The present invention discloses a cloud-client rendering computing method based on an adaptive virtualized rendering pipeline, comprising the following steps of: defining a rendering pipeline, including defining a rendering resource, a rendering algorithm, and a read-write relationship between the rendering algorithm and the rendering resource; selecting an optimal cloud-client computing distribution solution in a real-time manner from a cloud-client computing distribution solution set comprising each rendering resource that is allocated to a cloud or client for computing, based on self-defined optimization objectives and an optimization budget of a framework user; and executing a corresponding rendering algorithm on cloud and/or on a client according to the cloud-client computing distribution solution, thereby obtaining a rendering result. The rendering framework can adaptively select the cloud-client computing distribution solution upon cloud-client combined rendering and dynamically adjust it with a change of an optimization parameter.

Cloud-client rendering method based on adaptive virtualized rendering pipeline

Disclosed is a method for adapting a deep learning framework to a hardware device based on a unified backend engine, which comprises the following steps: S1, adding the unified backend engine to the deep learning framework; S2, adding the unified backend engine to the hardware device; S3, converting a computational graph, wherein the computational graph compiled and generated by the deep learning framework is converted into an intermediate representation of the unified backend engine; S4, compiling the intermediate representation, wherein the unified backend engine compiles the intermediate representation on the hardware device to generate an executable object; S5, running the executable object, wherein the deep learning framework runs the executable object on the hardware device; S6: managing memory of the unified backend engine.

Method for adapting deep learning framework to hardware device based on unified backend engine

The present disclosure discloses a method for execution of a computational graph in a neural network model and an apparatus thereof, including: creating task execution bodies on a native machine according to a physical computational graph compiled and generated by a deep learning framework, and designing a solution for allocating a plurality of idle memory blocks to each task execution body, so that the entire computational graph participates in deep learning training tasks of different batches of data in a pipelining and parallelizing manner.

Method for execution of computational graph in neural network model and apparatus thereof

The disclosure discloses a method of executing dynamic graph for neural network computation and the apparatus thereof. The method of executing dynamic graph includes the following steps: S: constructing and distributing an operator and a tensor; S: deducing an operator executing process by an operator interpreter; S: constructing an instruction of a virtual machine at runtime by the operator interpreter; S: sending the instruction to the virtual machine at runtime by the operator interpreter; S: scheduling the instruction by the virtual machine; and S: releasing an executed instruction by the virtual machine. According to the method of executing dynamic graph for neural network computation and the apparatus thereof provided by the disclosure, runtime is abstracted to be the virtual machine, and the virtual machine acquires a sub-graph of each step constructed by a user in real time through the interpreter and schedules, the virtual machines issues, and executes each sub-graph.

Method and apparatus of executing dynamic graph for neural network computation

The present invention discloses a color contrast enhanced rendering method, device and system suitable for an optical see-through head-mounted display. The method includes: (1) acquiring a background environment in real time to obtain a background video and performing Gaussian blur and visual field correction on the video; (2) converting an original rendering color and a processed video color from an RGB color space to a CIELAB color space scaled to a unit sphere range; (3) finding an optimal rendering color based on the original rendering color and the processed video color in the scaled CIELAB space according to a set color difference constraint, a chromaticity saturation constraint, a brightness constraint and a just noticeable difference constraint; and (4) after converting the optimal rendering color back to the RGB space, performing real-time rendering by using the optimal rendering color of the RGB space. The method is widely applied and capable of obviously improving the discrimination between a virtual content and a background environment and supporting various virtual scenes and background environments without a pre-preparation process.

Color contrast enhanced rendering method, device and system suitable for optical see-through head-mounted display

The disclosure discloses a method of neural network model computation-oriented intermediate representation and apparatus thereof. The method includes the following steps: S1, parsing an input model file so as to acquire topological structure information of a neural network; S2, constructing a logical computation graph; S21, inferring physical layout information of each operator in the logical computation graph; S22, inferring meta attributes of each operator in the logical computation graph; S23, inferring description information of input and output logical tensors of each operator in the logical computation graph; S3, constructing a physical computation graph; S31, generating a physical computation graph, etc.

Method of neural network model computation-oriented intermediate representation by constructing physical computation graph, inferring information of input and output tensor edges of each node therein, performing memory optimization on tensor edges, and optimizing physical computation graph

The present disclosure provides a neural network computing-oriented modeling method and apparatus for distributed data routing. The method includes the following steps: S1, designing the distributed attribute of a physical tensor: abstracting a mapping relationship between a logic tensor and the physical tensor into three distributed attributes including a broadcast attribute, a scatter attribute and a local reduction attribute; S2, deducing the distributed attribute of an output tensor: specifying the distributed attribute of an input tensor, and then deducing the legal distributed attribute of the output tensor according to the known distributed attribute of the input tensor; and S3, judging, according to the distributed attribute situation, whether an intermediate communication primitive needs to be inserted to obtain the distributed attribute of a local physical tensor. The difficulty of distributed design and development is low, and the development of application of a deep neural network large model is promoted.

Neural network computing-oriented modeling method and apparatus for distributed data routing

Growing community of inventors

Hangzhou, China

Hujun Bao