Vmware, Inc.

Jayneel Gandhi

Irina Calciu

Pratap Subrahmanyam

Aasheesh Kolli

Lalith Suresh

Rajesh Venkatasubramanian

Marcos Kawazoe Aguilera

Kiran Tati

Michael Wei

Xavier Deguillard

Nadav Amit

Amy Tai

Christopher John Rossbach

Gerd Zellweger

Timothy Merrifield

Sujay Yadalam Sudarshan

Stanko Novakovic

Pradeep Fernando

Reto Achermann

Systems and methods are described for communications across privilege domains within a central processing unit ('CPU') core. The CPU core can store a kernel context associated with an operating system within the CPU. An application can request access to the CPU, and the CPU can load a user context associated with the application into the CPU. The CPU can execute instructions from the application while both the kernel context and the user context persist in the CPU. Because both contexts are stored on the CPU, the CPU can switch contexts without loading or unloading context data from memory.

Communications across privilege domains within a central processing unit core

A device is connected via a coherence interconnect to a CPU with a cache. The device monitors cache coherence events via the coherence interconnect, where the cache coherence events relate to the cache of the CPU. The device also includes a buffer that can contain representations, such as addresses, of cache lines. If a coherence event occurs on the coherence interconnect indicating that a cache line in the CPU's cache is dirty, then the device is configured to add an entry to the buffer to record the dirty cache line.

Using cache coherent FPGAS to track dirty cache lines

An example method of managing memory in a computer system implementing non-uniform memory access (NUMA) by a plurality of sockets each having a processor component and a memory component is described. The method includes replicating page tables for an application executing on a first socket of the plurality of sockets across each of the plurality of sockets; associating metadata for pages of the memory storing the replicated page tables in each of the plurality of sockets; and updating the replicated page tables using the metadata to locate the pages of the memory that store the replicated page tables.

Transparent self-replicating page tables in computing systems

Disclosed are embodiments for migrating a virtual machine (VM) from a source host to a destination host while the virtual machine is running on the destination host. The system includes an RDMA facility connected between the source and destination hosts and a device coupled to a local memory, the local memory being responsible for memory pages of the VM instead of the source host. The device is configured to copy pages of the VM to the destination host and to maintain correct operation of the VM by monitoring coherence events, such as a cache miss, caused by the virtual machine running on the destination host. The device services these cache misses using the RDMA facility and copies the cache line satisfying the cache miss to the CPU running the VM. The device also tracks the cache misses to create an access pattern that it uses to predict future cache misses.

Using cache coherent FPGAS to accelerate post-copy migration

Disclosed is a method for performing write-back operations to maintain coherence of remote memories in a memory pool. When a local application makes a request for a page of memory that is in the memory pool but not local, a device obtains the page through its RDMA facility and thereafter keeps track of the page for any changes made by the application to the page by storing the page locally and monitoring cache coherency events of cache lines that make up the page. If a requested page become dirty, then periodically the dirty cache lines of the dirty page are written back to the remote memory from which the pages were obtained. In addition, all dirty cache lines are written back when the local memory storing the page becomes full or the application closes a region containing the page.

Using cache coherent FPGAS to accelerate remote memory write-back

Described herein is a method for tracking changes to memory locations made by an application. In one embodiment, the application decides to start tracking and sends a list of virtual memory pages to be tracked to an operating system via an interface. The operating system converts the list of virtual memory pages to a list of physical addresses and sends the list of physical addresses to a hardware unit which performs the tracking by detecting write backs on a coherence interconnect coupled to the hardware unit. After the application ends tracking, the application requests a list of dirty cache lines. In response to the request, the operating system obtains the list of dirty cache lines from the hardware unit and adds the list to a buffer that the application can read. In other embodiments, the operating system can perform the tracking without the application making the request.

Programming interfaces for accurate dirty data tracking

A virtual machine running on a source host is live migrated to a destination host. The source host includes a first processing node with a first processing hardware and a first memory, and a second processing node with a second processing hardware and a second memory. While the virtual machine is running on the first processing hardware, the second processing hardware tracks cache lines of the first processing hardware that become dirty as a result of write operations performed on one or more memory pages of the virtual machine. The dirty cache lines are copied to the destination host in units of a cache line or groups of cache lines.

Using cache coherent FPGAS to accelerate live migration of virtual machines

Disclosed is a technique in which an application can record changes it makes to physical memory. In the technique, the application specifies a virtual memory region which is converted to a plurality of cache lines, each of which is monitored for changes by a device connected to a coherence interconnect coupled to the processor caches. The application sends a start signal to start the logging process and an end signal to stop the process. During the logging process, when a change occurs to one of the cache lines, an undo entry corresponding to the change is created and entered into a transaction log residing in persistent memory. The transaction log containing the undo entries makes the set of changes recorded in the transaction log atomic. If a failure occurs, the recorded changes can be undone as if they never occurred.

Failure-atomic logging for persistent memory systems with cache-coherent FPGAs

Described herein is a method for tracking changes made by an application. Embodiments include determining, by a processor, a write-back of a cache line from a hardware unit associated with a socket of a plurality of sockets to a page table entry of a page table in a memory location associated with the processor. Embodiments include adding, by the processor, the cache line to a list of dirty cache lines. Embodiments include, for each respective cache line in the list of dirty cache lines, identifying, by the processor, a memory location associated with a respective socket of the plurality of sockets corresponding to the respective cache line and updating, by the processor, an entry of a page table replica at the memory location based on the respective cache line.

Accelerating replication of page tables for multi-socket machines

Techniques for efficiently managing the interruption of user-level critical sections are provided. In certain embodiments, a physical CPU of a computer system can execute a critical section of a user-level thread of an application, where program code for the critical section is marked with CPU instruction(s) indicating that the critical section should be executed atomically. The physical CPU can detect, while executing the critical section, an event to be handled by an OS kernel of the computer system and upon detecting the event, revert changes performed within the critical section. The physical CPU can then invoke a trap handler of the OS kernel, and in response the OS kernel can invoke a user-level handler of the application with information including (1) the identity of the user-level thread, (2) an indication of the event, (3) the physical CPU state upon detecting the event, and (4) an indication that the user-level thread was interrupted while in the critical section.

Efficiently managing the interruption of user-level critical sections

A module for rewriting application code as a failure-atomic transaction is disclosed. An application delineates a code fragment that is to become a failure-atomic transaction by writing a tx_begin( ) and tx_end( ) functions into its code. A module detects the tx_begin( ) and tx_end( ) functions and rewrites the code fragment as a failure-atomic transaction. The rewritten code is then available to be executed by the application. The rewritten code stores values and locations of stores to persistent memory locations so that the transaction can be either undone or redone in the case of an application error or a power failure. If the transaction is an undo type of transaction, the application can be rolled back to a state prior to the failure. If the transaction is a redo type of transaction, the application can be advanced to a correct state after the error.

Failure-atomic persistent memory logging using binary translation

Disclosed are embodiments for running an application on a local processor when the application is dependent on pages not locally present but contained in a remote host. The system is informed that the pages on which the application depends are locally present. While running, the application encounters a cache miss and a cache line satisfying the miss from the remote host is obtained and provided to the application. Alternatively, the page containing the cache line satisfying the miss is obtained and the portion of the page not including the cache line is stored locally while the cache line is provided to the application. The cache miss is discovered by monitoring coherence events on a coherence interconnect connected to the local processor. In some embodiments, the cache misses are tracked and provide a way to predict a set of pages to be pre-fetched in anticipation of the next cache misses.

Using cache coherent FPGAS to accelerate remote access

Exemplary methods, apparatuses, and systems include a distributed memory agent within a first node intercepting an operating system request to open a file from an application running on the first node. The request includes a file identifier, which the distributed memory agent transmits to a remote memory manager. The distributed memory agent receives, from the remote memory manager, a memory location within a second node for the file identifier and information to establish a remote direct memory access channel between the first node and the second node. In response to the request to open the file, the distributed memory agent establishes the remote direct memory access channel between the first node and the second node. The remote direct memory access channel allows the first node to read directly from or write directly to the memory location within the second node while bypassing an operating system of the second node.

File system interface for remote direct memory access

The disclosure provides an approach for tracking metadata (e.g., accessed and dirty bits) of page tables at finer granularity than the size of the page tables. A disclosed herein, modification to existing hardware design may enable finer page table granularity of metadata, leading to more precise representation of the state of memory and an improvement to system performance and efficiency. Finer grain dirty metadata can dramatically improve the efficiency and simplicity of subsystems.

Decoupling memory metadata granularity from page size

The disclosure provides an approach for atomically executing computer instructions by a CPU of a computing device comprising non-volatile memory, the CPU configured to implement hardware transactional memory (HTM). The approach generally includes reading an instruction within a section of code designated as an HTM transaction, determining whether the instruction causes a data conflict with another thread, and copying cache lines from memory into a cache of the CPU. The approach further includes marking the copied cache lines as transactional, processing the instruction to create a persistent log within non-volatile memory, and unmarking the copied cache lines from transactional, to non-transactional.

Growing community of inventors

Sunnyvale, CA, United States of America

Jayneel Gandhi