Hewlett Packard Enterprise Development LP

Aditya Dhakal

Dejan S Milojicic

Gourav Rattihalli

Puneet Sharma

Sai Rahul Chalamalasetti

Diman Zad Tootaghaj

Tobias Pfandzelter

Kaiwen Cao

Yunming Xiao

Ninad Sanjay Hogade

Pedro H R Bruel

Eitan Frachtenberg

Rolando Pablo Hong Enriquez

Systems and methods are provided for a dynamic communication architecture that can include decentralized heterogenous accelerators, and an operation of mapping corresponding decentralized virtualized accelerators. The mapping of corresponding decentralized virtualized accelerators is performed on top of, for example, a memory-mapped multi-accelerator communication architecture to enable effective and flexible sharing of data. The mapping can be dynamically adjusted with regard to the workload running by analyzing workflow communication.

Dynamic communication architecture for decentralized heterogenous accelerators

Systems and methods are provided for implementing an iterative function deployment in a Function-as-a-Service (FaaS) computing environment. For example, the system may receive a request to execute a program and convert machine-readable code from the request into functions that are deployable in the FaaS computing environment. The system may determine a first deployment of the functions that defines scheduling and execution tasks, and also an execution metric by a trained prediction model. The system may adjust execution of the plurality of functions and also adjust the execution metric in a second deployment. The system may implement the second deployment at run-time, such that the machine-readable code from the request is executed by the computing components of the FaaS infrastructure in accordance with the second deployment.

Automated program conversion for heterogeneous and FaaS computing with minimal human intervention

A first NIC monitors a key-value cache associated with an LLM executed by a compute node that includes the first NIC and an accelerator. The key-value cache is stored in a memory associated with the accelerator. Responsive to detecting that the key-value cache is updated by the accelerator, the first NIC transfers a copy of the key-value cache update to a remote storage node. The key-value cache is deleted from the memory after the query is inferred. Responsive to receiving a follow-up user query, the first NIC determines a storage location on the remote storage node that stores the key-value cache corresponding to the user query and sends a KV-cache-transfer request to a second NIC on the remote storage node, the KV-cache-transfer request specifying the storage location, thereby facilitating the second NIC to transfer the key-value cache corresponding to the user query from the specified storage location to the memory.

Efficient key-value cache management for large language models

In certain embodiments, a method includes receiving, at an interface of a Smart network interface card (SmartNIC) of a computing device, via a network, a network data unit; processing, by a data allocator of a SmartNIC subsystem of the SmartNIC, the network data unit to make a determination that data included in the network data unit is intended for processing by an accelerator of the computing device, wherein the accelerator is configured to execute a machine learning algorithm; storing, by the data allocator and based on the determination, the data in a local buffer of the SmartNIC subsystem; identifying, by the data allocator, a memory resource associated with the accelerator; and transferring the data from the local buffer to the memory resource.

Smart network interface card control plane for distributed machine learning workloads

Growing community of inventors

Santa Clarita, CA, United States of America

Aditya Dhakal