Vmware, Inc.

Aasheesh Kolli

Irina Calciu

Jayneel Gandhi

Pratap Subrahmanyam

Vijaychidambaram Velayudhan Pillai

Pradeep Fernando

A device is connected via a coherence interconnect to a CPU with a cache. The device monitors cache coherence events via the coherence interconnect, where the cache coherence events relate to the cache of the CPU. The device also includes a buffer that can contain representations, such as addresses, of cache lines. If a coherence event occurs on the coherence interconnect indicating that a cache line in the CPU's cache is dirty, then the device is configured to add an entry to the buffer to record the dirty cache line.

Using cache coherent FPGAS to track dirty cache lines

Disclosed are embodiments for migrating a virtual machine (VM) from a source host to a destination host while the virtual machine is running on the destination host. The system includes an RDMA facility connected between the source and destination hosts and a device coupled to a local memory, the local memory being responsible for memory pages of the VM instead of the source host. The device is configured to copy pages of the VM to the destination host and to maintain correct operation of the VM by monitoring coherence events, such as a cache miss, caused by the virtual machine running on the destination host. The device services these cache misses using the RDMA facility and copies the cache line satisfying the cache miss to the CPU running the VM. The device also tracks the cache misses to create an access pattern that it uses to predict future cache misses.

Using cache coherent FPGAS to accelerate post-copy migration

The disclosure provides an approach for testing if a cache line of a cache has been flushed to non-volatile memory (NVM). The approach generally includes reading, by a central processing unit (CPU), data from the NVM. The approach further includes storing, by the CPU, a copy of the data in the cache as a cache line. The approach further includes modifying, by the CPU, at least a portion of the copy of the data in the cache. The approach further includes requesting, by the CPU, the cache line be flushed to the NVM. The approach further includes performing, by the CPU, one or more instructions in parallel to the cache line being flushed to the NVM. The approach further includes requesting, by the CPU, a state of the cache line and determining if the cache line has been persisted in the NVM based on the state of the cache line.

Cache line persistence indicator for non-volatile memory using coherence states

Disclosed is a method for performing write-back operations to maintain coherence of remote memories in a memory pool. When a local application makes a request for a page of memory that is in the memory pool but not local, a device obtains the page through its RDMA facility and thereafter keeps track of the page for any changes made by the application to the page by storing the page locally and monitoring cache coherency events of cache lines that make up the page. If a requested page become dirty, then periodically the dirty cache lines of the dirty page are written back to the remote memory from which the pages were obtained. In addition, all dirty cache lines are written back when the local memory storing the page becomes full or the application closes a region containing the page.

Using cache coherent FPGAS to accelerate remote memory write-back

Described herein is a method for tracking changes to memory locations made by an application. In one embodiment, the application decides to start tracking and sends a list of virtual memory pages to be tracked to an operating system via an interface. The operating system converts the list of virtual memory pages to a list of physical addresses and sends the list of physical addresses to a hardware unit which performs the tracking by detecting write backs on a coherence interconnect coupled to the hardware unit. After the application ends tracking, the application requests a list of dirty cache lines. In response to the request, the operating system obtains the list of dirty cache lines from the hardware unit and adds the list to a buffer that the application can read. In other embodiments, the operating system can perform the tracking without the application making the request.

Programming interfaces for accurate dirty data tracking

A virtual machine running on a source host is live migrated to a destination host. The source host includes a first processing node with a first processing hardware and a first memory, and a second processing node with a second processing hardware and a second memory. While the virtual machine is running on the first processing hardware, the second processing hardware tracks cache lines of the first processing hardware that become dirty as a result of write operations performed on one or more memory pages of the virtual machine. The dirty cache lines are copied to the destination host in units of a cache line or groups of cache lines.

Using cache coherent FPGAS to accelerate live migration of virtual machines

An example method of managing persistent memory (PM) in a computing system includes: issuing, by an application executing in the computing system, store instructions to an address space of the application, the address space including a region mapped to the PM; recording, by a central processing unit (CPU) in the computing system, cache line addresses in a log, the cache line addresses corresponding to cache lines in the address space of the application targeted by the store instructions; and issuing, by the application, one or more instructions to flush cache lines from cache of the CPU identified by the cache line addresses in the log.

Dirty data tracking in persistent memory systems

Disclosed is a technique in which an application can record changes it makes to physical memory. In the technique, the application specifies a virtual memory region which is converted to a plurality of cache lines, each of which is monitored for changes by a device connected to a coherence interconnect coupled to the processor caches. The application sends a start signal to start the logging process and an end signal to stop the process. During the logging process, when a change occurs to one of the cache lines, an undo entry corresponding to the change is created and entered into a transaction log residing in persistent memory. The transaction log containing the undo entries makes the set of changes recorded in the transaction log atomic. If a failure occurs, the recorded changes can be undone as if they never occurred.

Failure-atomic logging for persistent memory systems with cache-coherent FPGAs

Described herein is a method for tracking changes made by an application. Embodiments include determining, by a processor, a write-back of a cache line from a hardware unit associated with a socket of a plurality of sockets to a page table entry of a page table in a memory location associated with the processor. Embodiments include adding, by the processor, the cache line to a list of dirty cache lines. Embodiments include, for each respective cache line in the list of dirty cache lines, identifying, by the processor, a memory location associated with a respective socket of the plurality of sockets corresponding to the respective cache line and updating, by the processor, an entry of a page table replica at the memory location based on the respective cache line.

Accelerating replication of page tables for multi-socket machines

An example method of configuring an application to manage persistent memory (PM) in a computer system includes: modifying, by a compiler during compilation of the application, source code of the application to add instructions to update tracking metadata for store instructions in the source code that target memory blocks mapped to the PM; compiling, by the compiler, the source code to generate an executable process; and issuing, by a synchronization routine executing on the computer, write-back instructions during execution of the executable process based on the tracking metadata.

A module for rewriting application code as a failure-atomic transaction is disclosed. An application delineates a code fragment that is to become a failure-atomic transaction by writing a tx_begin( ) and tx_end( ) functions into its code. A module detects the tx_begin( ) and tx_end( ) functions and rewrites the code fragment as a failure-atomic transaction. The rewritten code is then available to be executed by the application. The rewritten code stores values and locations of stores to persistent memory locations so that the transaction can be either undone or redone in the case of an application error or a power failure. If the transaction is an undo type of transaction, the application can be rolled back to a state prior to the failure. If the transaction is a redo type of transaction, the application can be advanced to a correct state after the error.

Failure-atomic persistent memory logging using binary translation

Disclosed are embodiments for running an application on a local processor when the application is dependent on pages not locally present but contained in a remote host. The system is informed that the pages on which the application depends are locally present. While running, the application encounters a cache miss and a cache line satisfying the miss from the remote host is obtained and provided to the application. Alternatively, the page containing the cache line satisfying the miss is obtained and the portion of the page not including the cache line is stored locally while the cache line is provided to the application. The cache miss is discovered by monitoring coherence events on a coherence interconnect connected to the local processor. In some embodiments, the cache misses are tracked and provide a way to predict a set of pages to be pre-fetched in anticipation of the next cache misses.

Using cache coherent FPGAS to accelerate remote access

The disclosure provides an approach for atomically executing computer instructions by a CPU of a computing device comprising non-volatile memory, the CPU configured to implement hardware transactional memory (HTM). The approach generally includes reading an instruction within a section of code designated as an HTM transaction, determining whether the instruction causes a data conflict with another thread, and copying cache lines from memory into a cache of the CPU. The approach further includes marking the copied cache lines as transactional, processing the instruction to create a persistent log within non-volatile memory, and unmarking the copied cache lines from transactional, to non-transactional.

Growing community of inventors

Palo Alto, CA, United States of America

Aasheesh Kolli