Alibaba Group Holding Limited

Weifeng Zhang

Guoyang Chen

Yuan Xie

Yu Pu

Minxuan Zhou

Yongzhi Zhang

Hongzhong Zheng

Dimin Niu

Fei Sun

Shuangchen Li

Yingmin Li

Jingjun Chu

Xipeng Shen

The present disclosure relates to a processing in memory (PIM) enabled device for executing a neural network model. The PIM enabled device comprises a memory block assembly comprising a first array of memory blocks, a second array of memory blocks adjacent to the first array of memory blocks, a plurality of first data links associated with the first array of memory blocks and the second array of memory blocks, wherein each data link of the plurality of first data links communicatively couples two corresponding memory blocks of which are from the first array of memory blocks and the second array of memory blocks respectively, and a second data link communicatively coupled to the plurality of first data links. The data from a first memory block of the first array of memory blocks can be transferable to a second memory block of the second array of memory blocks via the plurality of first data links and the second data link.

Data layout conscious processing in memory architecture for executing neural network model

Methods and devices, the method including receiving a matrix of a neural network model; classifying at least a portion of the matrix as a first section based on a first distribution pattern of non-zero elements of the portion of the matrix; and identifying memory addresses of the non-zero elements in the first section of the matrix for loading, according to a first order determined based on the first distribution pattern, the non-zero elements in the first section into one or more vector registers.

Method and device for matrix multiplication optimization using vector registers

Methods and apparatus for reducing a size of a neural network model, the method including: compressing data of the neural network model; identifying structure information of a vector register, wherein the structure information includes a number of registers included in the vector register; comparing a number of elements in the compressed data with a first condition, wherein the first condition is determined based on the number of registers in the vector register; and in response to the number of elements satisfying the first condition, associating the compressed data with the vector register to enable loading the compressed data to the vector register.

Method and device for reducing a size of a neural network model

The present disclosure relates to a method for scheduling a computation graph on a processing in memory (PIM) enabled device comprising a memory block assembly. The method comprises allocating a first node of the computation graph on a first memory block of a first array of memory blocks in the memory block assembly and allocating a second node of the computation graph on a second memory block of a second array of memory blocks in the memory block assembly, wherein output data of the first node is used for executing the second node. The memory block assembly can be configured to support data transfer from the first memory block to the second memory block via an internal data coupling in the memory block assembly.

Data layout optimization on processing in memory architecture for executing neural network model

The present disclosure relates to a method for scheduling a computation graph on heterogeneous computing resources. The method comprises generating an augmented computation graph that includes a first set of replica nodes corresponding to a first node in the computation graph and a second set of replica nodes corresponding to a second node in the computation graph, wherein the replica nodes of the first set are connected by edges to the replica nodes of the second set according to dependency between the first node and the second node in the computation graph, adapting the augmented computation graph to include performance values for the edges, the replica nodes of the first set, and the replica nodes of the second set, and determining a path across the adapted computation graph via one replica node of the first set and one replica node of the second set based on the performance values.

Computation graph mapping in heterogeneous computer system

A method for quantizing a machine learning model during an inference phase, including determining a normalization factor using a set of floating-point values and a damped value of a damped value sequence; and assigning a quantized value for each floating-point value of the set of floating-point values based on the damped value sequence and the normalization factor.

Quantizing machine learning models with balanced resolution via damped encoding

Embodiments of the disclosure provide systems and methods for allocating memory space in a memory device. The system can include: a memory device for providing the memory space; and a compiler component configured for: receiving a request for allocating a data array having a plurality of data elements in the memory device, wherein each of the plurality of data elements has a logical address; generating an instruction for allocating memory space for the data array in the memory device based on the request; generating device addresses for the plurality of data elements in the memory device based on logical addresses of the plurality of data elements; and allocating the memory space for the data array in the memory device based on the device addresses and the instruction.

System and method for allocating memory space

Embodiments of the present disclosure introduce an offset-based representation for a computation graph of a neural network model. In this representation, the edges in the graph are implemented as offsets to the base address of the graph when the model is loaded into memory rather than pointers to address locations in memory. By replacing the pointer fields with offsets, the model in memory is no longer dependent on the allocation of memory. This overcomes the limitations of performance and/or power that is critical for comparing two models and copying a model to another memory location in latency sensitive applications and in mobile environments.

Efficient in-memory representation of computation graph for fast serialization and comparison

The present disclosure relates to a method for detecting error in executing a computation graph on heterogeneous computing devices. The method comprises receiving a first reference value as an execution result for a first node of the computation graph from a reference device included in the heterogeneous computing devices, receiving a first target value from a target device included in the heterogeneous computing devices as an execution result by the target device for the first node, comparing the first reference value and the first target value, and determining whether the first target value is in error based on the comparison of the first reference value and the first target value. The method can further comprise generating multiple execution contexts for executing the computation graph on the heterogeneous computing devices.

Growing community of inventors

San Mateo, CA, United States of America

Weifeng Zhang