Multi-node cluster-based inference method and system through gpu separate allocation of pre-trained layer and fine-tuning layer of multiple deep learning models

Abstract

A multi-node cluster-based inference method through GPU separation allocation of a pre-trained layer and a fine-tuning layer of multiple deep learning models. The method includes: receiving an input value from a client; distributing the received input value, and transmitting the first input value to a first computation node including a container in which a neural network bundle of a first stage is loaded; performing, by a first container of the first computation nodes, an operation through a neural network layer of a GPU by using the received first input value as an input, and generating a first output value; and selecting a container in which the neural network bundle of the next stage is loaded, and transmitting the second output value to the computation node that includes the container in which the next stage is to be executed or the container in which to execute the next stage.