Oracle International Corporation

Suman Mallapura Somasundar

Vishal Vishnoi

Xin Xu

Stevan Malesevic

Ping L Lin

Xinwei Zhang

Techniques are disclosed for providing a scalable multi-tenant serve pool for chatbot systems. A query serving system (QSS) receives a request to serve a query for a new skillbot. The QSS comprises a plurality of deployments, each of which is configured to host a plurality of machine-learning models, each machine-learning model being associated with a skillbot, each deployment including a serving container and a model manager container that hosts a model manager, the serving container including a plurality of sub-containers, each of which hosts one of the machine-learning models downloaded by the model manager. The QSS selects a first deployment to be assigned to the new skillbot based on a first criterion, and loads the machine-learning model associated with the new skillbot into the first deployment. The machine-learning model is trained to serve the query for the new skillbot. The query is served using the machine-learning model.

Fast and scalable multi-tenant serve pool for chatbots

Techniques are disclosed for providing a scalable multi-tenant serve pool for chatbot systems. A query serving system (QSS) receives a request to serve a query for a skillbot. The QSS includes: (i) a plurality of deployments in a serving pool, and (ii) a plurality of deployments in a free pool. The QSS determines whether a first deployment from the plurality of deployments in the serving pool can serve the query based on an identifier of the skillbot. In response to determining that the first deployment cannot serve the query, the QSS selects a second deployment from the plurality of deployments in the free pool to be assigned to the skillbot, and loads a machine-learning model associated with the skillbot into the second deployment, wherein the machine-learning model is trained to serve the query for the skillbot. The query is served using the machine-learning model loaded into the second deployment.

A training request including an identifier that is indicative of a type of a machine learning (ML) model that is to be trained is received. A plurality of workers are maintained in a training pool, and a plurality of jobs are maintained in a queue of training jobs. Each worker is configured to train a particular type of ML model. Upon the training request being validated, a training job is created for the request and submitted to the queue of training jobs. For each type of ML model, a first metric and a second metric is obtained. A target metric is computed based on the first and the second metrics. The number of workers included in the training pool is modified based on the target metric.

Growing community of inventors

Sunnyvale, CA, United States of America

Suman Mallapura Somasundar