Vmware, Inc.

Nicira, Inc.

Wenyi Jiang

Boon Seong Ang

Guolin Yang

Rajeev Nair

Aditya Gajanana Holla

Yong Wang

Jin Heo

Dexiang Wang

Rishi Mehta

Srikar Tati

Bin Qian

Daniel G Wing

Srividya Murali

Ying Gross

Chidambareswaran Raman

Dileep Kumar Devireddy

Subin Cyriac Mathew

Alexander Tessmer

Jingchun Jason Jiang

Ayyappan Veeraiyan

Shilpi Agarwal

Shrikrishna Khare

Zongyun Lai

Kairav Padarthy

Subbarao Narahari

Jayant Jain

Ganesan Chandrashekhar

Donghai Han

Peng T Li

Jianjun Shen

Xinhua Hong

Jia Yu

Ankur Kumar Sharma

Aditya Krishna Sonthy

Aditya G Holla

Some embodiments of the invention provide a method of migrating a virtual machine (VM) from a first host computer to a second host computer, the first host computer having a first PNIC, the second host computer having a second PNIC, the first and second PNICs for performing at least one of network forwarding operations and middlebox service operations for the VM. At an RDMA client executing on a set of one or more processors of the second PNIC, the method receives a notification from the second host computer indicating a data migration that is performed to migrate the particular VM from the first host computer to the second host computer has started. Based on the notification, at the RDMA client, the method directs an RDMA server executing on the first PNIC to provide networking state data associated with at least one of network forwarding operations and middlebox service operations that the first PNIC performs for the VM. The provided networking state data resides in a memory of the first PNIC that is accessible to the RDMA server.

Live workload migration for hardware offload capable NICs using RDMA

An approach for a dynamic provisioning of multiple RSS engines is provided. In an embodiment, a method comprises monitoring a CPU usage of hardware queues implemented in a plurality of RSS pools, and determining whether a CPU usage of any hardware queue, implemented in a particular RSS pool of the plurality of RSS pools, has increased above a threshold value. In response to determining that a CPU usage of a particular hardware queue, implemented in the particular RSS pool, has increased above the threshold value, it is determined whether the particular RSS pool includes an unused hardware queue (a queue with light CPU usage). If such an unused hardware queue is presented, then an indirection table that is associated with the particular RSS pool is modified to remap one or more data flows from the particular hardware queue to the unused hardware queue.

Dynamic provisioning of multiple RSS engines

Some embodiments provide a method for a first smart NIC of multiple smart NICs of a host computer. Each of the smart NICs executes a smart NIC operating system that performs virtual networking operations for a set of data compute machines executing on the host computer. The method receives a data message sent by one of the data compute machines executing on the host computer. The method performs virtual networking operations on the data message to determine that the data message is to be transmitted from a port of a second smart NIC of the multiple smart NICs. The method passes the data message to the second smart NIC via a private communication channel connecting the plurality of smart NICs.

Teaming of smart NICs

A first host receives a packet from a first compute node for a second compute node of a second host. The payload is larger than a maximum transmission unit size. The first packet is encapsulated with an outer header. The first host analyzes a length of at least a portion of the outer header in determining a size of an encrypted segment of the payload. Then, the first host forms a plurality of packets where each packet in the packets includes an encrypted segment of the payload, a respective encryption header, and a respective authentication value. The payload of the first packet is segmented to form a plurality of encrypted segments based on the size. The first host sends the packets to the second host and receives an indication that a packet was not received. A second packet including the encrypted segment is sent to the second compute node.

Segmentation of encrypted segments in networks

Some embodiments of the invention provide a method for configuring a physical network card or physical network controller (pNIC) to provide flow processing offload (FPO) for a host computer connected to the pNIC. The host computers host a set of compute nodes in a virtual network. The set of compute nodes are each associated with a set of interfaces that are each assigned a locally-unique virtual port identifier (VPID) by a flow processing and action generator. The pNIC includes a set of interfaces that are assigned physical port identifiers (PPIDs) by the pNIC. The method includes providing the pNIC with a set of mappings between VPIDs and PPIDs. The method also includes sending updates to the mappings as compute nodes migrate, connect to different interfaces of the pNIC, are assigned different VPIDs, etc. In some embodiments, the flow processing and action generator executes on processing units of the host computer, while in other embodiments, the flow processing and action generator executes on a set of processing units of a pNIC that includes flow processing hardware and a set of programmable processing units.

Configuring pNIC to perform flow processing offload using virtual port identifiers

In some embodiments, a method fragments a first packet into a plurality of fragments when a length of an encapsulated first packet is larger than a maximum transmission unit size. For each fragment in the plurality of fragments, fragmentation information is generated. The method encapsulates each fragment in the plurality of fragments with an outer header to form a plurality of encapsulated packets. The respective fragmentation information for each fragment is inserted in a portion of the outer header that is processed by endpoints of an overlay tunnel and not processed by a device along a path of the overlay tunnel. The plurality of encapsulated packets are sent via the overlay tunnel.

Packet fragmentation using outer header

Some embodiments of the invention provide a method for configuring multiple hardware offload units of a host computer to perform operations on packets associated with machines (e.g., virtual machines or containers) executing on the host computer and to pass the packets between each other efficiently. For instance, in some embodiments, the method configures a program executing on the host computer to identify a first hardware offload unit that has to perform a first operation on a packet associated with a particular machine and to provide the packet to the first hardware offload unit. The packet in some embodiments is a packet that the particular machine has sent to a destination machine on the network, or is a packet received from a source machine through a network and destined to the particular machine.

Packet processing with hardware offload units

Some embodiments provide a method for synchronizing state between multiple smart NICs of a host computer that perform operations using dynamic state information. At a first smart NIC of the plurality of smart NICs, the method stores a set of dynamic state information. The method synchronizes the set of dynamic state information across a communication channel that connects the smart NICs so that each of the smart NICs also stores the set of dynamic state information.

State sharing between smart NICs

In some embodiments, a method determines when a packet is fragmented into multiple fragmented packets in a flow between a first workload and a second workload. The method switches from generating an outer source port in the outer header using layer 4 information from the inner header to using layer 3 information from the inner header. A fragmented packet is encapsulated with the outer header that includes an outer source port value that is generated using the layer 3 information. The method initiates a process to determine when to switch back to using layer 4 information from the inner header to generate the outer source port. When it is determined to switch back to using layer 4 information, the method switches back to using layer 4 information from the inner header to generate the source port in the outer header of a packet from the first workload.

Adaptive source port generation in overlay networks

Some embodiments provide a method for a first smart NIC of multiple smart NICs of a host computer. Each of the smart NICs is for performing virtual networking operations for a set of data compute machines executing on the host computer. The method determines that the first smart NIC is elected to communicate with a network management and control system that configures the virtual networking operations. The method receives a set of configuration data for the virtual networking operations from the network management and control system. The method provides the received set of configuration data to the other smart NICs of the host computer.

Smart NIC leader election

Some embodiments provide a method for selecting a transmit queue of a network interface card (NIC) of a host computer for an outbound data message. The NIC includes multiple transmit queues and multiple receive queues. Each of the transmit queues is individually associated with a different receive queue, and the MC performs a load balancing operation to distribute inbound data messages among multiple receive queues. The method extracts a set of header values from a header of the outbound data message. The method uses the extracted set of header values to identify a receive queue which the MC would select for a corresponding inbound data message upon which the NIC performed the load balancing operation. The method selects a transmit queue associated with the identified receive queue to process the outbound data message.

Selection of paired transmit queue

Example methods and systems for logical network packet handling are described. In one example, a physical network interface controller (PNIC) may receive an egress packet associated with a packet flow via a first virtual function supported by the PNIC. The PNIC may steer the egress packet towards a processing pipeline by applying a filter associated with the first virtual function or content of the egress packet, or both. The egress packet may be processed using the processing pipeline to generate a processed packet by (a) retrieving a logical network policy associated with the packet flow from a datastore on the PNIC and (b) performing one or more actions according to the logical network policy. The processed packet may be forwarded towards the destination via a second virtual function supported by the PNIC or a physical network connected to the PNIC.

Logical network packet handling on physical network interface controller (PNIC)

Configuring PNIC to perform flow processing offload using virtual port identifiers

Example methods and systems for logical network packet handling are described. In one example, a physical network interface controller (PNIC) may receive an ingress encapsulated packet associated with a packet flow via a physical network. The ingress encapsulated packet may include an outer header and an inner packet that is destined for a virtualized computing instance. The ingress encapsulated packet may be steered towards a processing pipeline for processing to generate a processed packet. The processing pipeline may include (a) retrieving a logical network policy associated with the packet flow from a datastore on the PNIC; and (b) performing decapsulation to remove the outer header and one or more actions on the inner packet according to the logical network policy. The processed packet may be forwarded towards the virtualized computing instance via a virtual function supported by the PNIC or a physical network connected to the PNIC.

Example methods and systems for a programmable virtual network interface controller (VNIC) to perform packet processing are described. In one example, the programmable VNIC may modify a packet processing pipeline based on the instruction. The modification may include injecting a second packet processing stage among the multiple first packet processing stages of the packet processing pipeline. In response to detecting an ingress packet that requires processing by the programmable VNIC, the ingress packet may be steered towards the modified packet processing pipeline. The ingress packet may then be processed using the modified packet processing pipeline by performing the second packet processing stage (a) to bypass at least one of the multiple first processing stages, or (b) in addition to the multiple first processing stages.

Growing community of inventors

Palo Alto, CA, United States of America

Wenyi Jiang