Shanghaitech University

Shu Yin

Tianyuan Wu

Tongliang Deng

Yuanhao Li

Deep neural network checkpoint optimization system and method based on non-volatile memory are provided, where the client module and the server module register corresponding network structures in non-volatile memory and create data indexes and data communication protocols based on remote direct memory access (RDMA) before the start of training deep neural networks, and during the neural network training process, provide zero-copy, asynchronous, end-to-end neural network data persistence, which allows users to perform fine-grained checkpointing to ensure fault tolerance and data persistence without affecting the training speed.

Deep neural network checkpoint optimization system and method based on non-volatile memory

A middleware system and method for optimizing reading and writing of scientific data files; the method comprises: receiving data writing requests from users, constructing spatial grid based on the regional mask information of written scientific data, and merging grid data blocks of same region to construct the continuous non-equal width data blocks; grouping the continuous non-equal-width data blocks based on the regional mask information of the continuous non-equal-width data blocks to obtain the grouped non-equal-width data blocks containing regional grouping information, and encoding and indexing the grouped non-equal-width data blocks to obtain the encoded non-equal-width data blocks containing indexing information; retrieving corresponding data blocks and issuing the data blocks reading instructions based on regional reading information of the users; handling the reading and writing of metadata corresponding to scientific data; managing the reading and writing of the encoded non-equal width data blocks.

Middleware system and method for optimizing reading and writing of scientific data files

This application provides a data management system, method, terminal, and medium based on hybrid storage. The data management system includes: a first file system mount module, to mount at least one user-mode file system; a second file system mount module, to mount at least two independent back-end file systems based on the user-mode file system for storing hot data and cold data respectively; a data label module, to label the hot or cold attribute of the data in a user data request; a file system selection module, to store the data in the corresponding back-end file system and/or take the data out from the corresponding back-end file system according to the hot or cold attribute of the data.

Growing community of inventors

Shanghai, China

Shu Yin