Dell Products Inc.

Ro Monserrat

Jonathan I Krasner

Jerome J Cartmell

Thomas Mackintosh

Michael J Scharland

James Marriot Guyer

Scott Rowlands

Julie Zhivich

A pair of compute nodes, each having a separate PCIe root complex, are interconnected by a PCIe Non-Transparent Bridge (NTB). An instance of a NTB monitoring process is started for each root complex, and the CPU affinity of the NTB monitoring processes are set to cause each NTB monitoring process to be executed on CPU resources of each respective CPU root complex. The NTB monitoring process on a given root complex is allowed to sleep until a triggering event occurs that causes the NTB monitoring process to wake and determine the state of the NTB. One such triggering event is a failure of an atomicity algorithm on the compute node to obtain a lock on peer memory in connection with implementing an atomic read operation on the peer memory over the NTB.

Method and apparatus for monitoring a PCIe NTB

A deterministic hardware indictment process is used to indict one Compute Node (CN) of a pair of CNs in response to occurrence of a fatal error on a Peripheral Component Interconnect Express (PCIe) Non-Transparent Bridge (NTB) interconnecting the two CN. Status information is exchanged between the CNs on a communication medium that is separate from the NTB. The indictment process is run locally on each CN based on whatever information is available to the CN. When the CN does not receive status information from the other CN, the CN runs a self-indictment check. When status information is received, and only one CN reports an error, the indictment process indicts the compute node that reported the error. If both CNs report errors, an error severity comparison process is used to select a CN to be indicted. If the reported errors are equally severe, a default CN is indicted.

Deterministic hardware indictment in response to fatal PCIe ntb error

A storage engine has a pair of compute nodes, each compute node having a separate PCIe root complex and attached memory. The PCIe root complexes are interconnected by multiple Non-Transparent Bridge (NTB) links. The NTB resources are unequally shared, such that host IO devices are required to use a first subset of the NTB links to implement memory access operations on the memory of the peer compute node, whereas storage software memory access operations are able to be implemented on all of the NTB links. A NTB link arbitration system arbitrates usage of the first and second subsets of NTB links by the storage software, to distribute subsets of the storage software memory access operations on peer memory to the first and second subsets of NTB links, while causing all host IO device memory access operations on peer memory to be implemented on the first set of NTB links.

Non-transparent bridge selection

A host IO devices directly implements host read operations on both local memory, and on peer memory via a PCIe non-transparent bridge. When a host read operation is received by a host IO device from a host, the host IO device uses an API to obtain the physical address of the requested data on the peer memory, and generates a PCIe Transaction Layer Packet (TLP) addressed to the address in the peer memory. The TLP addressed to an address in the peer memory is passed over the NTB to the peer compute node to retrieve the data stored in the addressed slot of peer memory. The requested data is returned to the host IO device over the NTB, stored in a buffer, and read out to the host to directy respond to the host read operation.

Growing community of inventors

Medfield, MA, United States of America

Ro Monserrat