Oracle International Corporation

Stevan Malesevic

Vishal Vishnoi

Xin Xu

Suman Mallapura Somasundar

Techniques are disclosed for providing a scalable multi-tenant serve pool for chatbot systems. A query serving system (QSS) receives a request to serve a query for a new skillbot. The QSS comprises a plurality of deployments, each of which is configured to host a plurality of machine-learning models, each machine-learning model being associated with a skillbot, each deployment including a serving container and a model manager container that hosts a model manager, the serving container including a plurality of sub-containers, each of which hosts one of the machine-learning models downloaded by the model manager. The QSS selects a first deployment to be assigned to the new skillbot based on a first criterion, and loads the machine-learning model associated with the new skillbot into the first deployment. The machine-learning model is trained to serve the query for the new skillbot. The query is served using the machine-learning model.

Fast and scalable multi-tenant serve pool for chatbots

Techniques are disclosed for providing a scalable multi-tenant serve pool for chatbot systems. A query serving system (QSS) receives a request to serve a query for a skillbot. The QSS includes: (i) a plurality of deployments in a serving pool, and (ii) a plurality of deployments in a free pool. The QSS determines whether a first deployment from the plurality of deployments in the serving pool can serve the query based on an identifier of the skillbot. In response to determining that the first deployment cannot serve the query, the QSS selects a second deployment from the plurality of deployments in the free pool to be assigned to the skillbot, and loads a machine-learning model associated with the skillbot into the second deployment, wherein the machine-learning model is trained to serve the query for the skillbot. The query is served using the machine-learning model loaded into the second deployment.

Growing community of inventors

Glen Ellyn, IL, United States of America

Stevan Malesevic