Hewlett Packard Enterprise Development LP

Gourav Rattihalli

Dejan S Milojicic

Aditya Dhakal

Sai Rahul Chalamalasetti

Rolando Pablo Hong Enriquez

Tobias Pfandzelter

Eitan Frachtenberg

Ninad Sanjay Hogade

Pedro H R Bruel

Systems and methods are provided for implementing an iterative function deployment in a Function-as-a-Service (FaaS) computing environment. For example, the system may receive a request to execute a program and convert machine-readable code from the request into functions that are deployable in the FaaS computing environment. The system may determine a first deployment of the functions that defines scheduling and execution tasks, and also an execution metric by a trained prediction model. The system may adjust execution of the plurality of functions and also adjust the execution metric in a second deployment. The system may implement the second deployment at run-time, such that the machine-readable code from the request is executed by the computing components of the FaaS infrastructure in accordance with the second deployment.

Automated program conversion for heterogeneous and FaaS computing with minimal human intervention

A first NIC monitors a key-value cache associated with an LLM executed by a compute node that includes the first NIC and an accelerator. The key-value cache is stored in a memory associated with the accelerator. Responsive to detecting that the key-value cache is updated by the accelerator, the first NIC transfers a copy of the key-value cache update to a remote storage node. The key-value cache is deleted from the memory after the query is inferred. Responsive to receiving a follow-up user query, the first NIC determines a storage location on the remote storage node that stores the key-value cache corresponding to the user query and sends a KV-cache-transfer request to a second NIC on the remote storage node, the KV-cache-transfer request specifying the storage location, thereby facilitating the second NIC to transfer the key-value cache corresponding to the user query from the specified storage location to the memory.

Efficient key-value cache management for large language models

Growing community of inventors

Milpitas, CA, United States of America

Gourav Rattihalli