Vmware, Inc.

Advanced Micro Devices Corporation

Datrium, Inc.

Min Xu

Boris Weissman

Ganesh Venkitachalam

Vyacheslav Vladimirovich Malyugin

Daniel J Scales

Jeffrey W Sheldon

Ming-Ju Edward Lee

Arvind Bomdica

Rohit Jain

Alexander Klaiber

Peter Chen

Matthew David Ginzton

Yikai Liang

Petr Vandrovec

Xin Liu

Samudyatha Suryanarayana

Gayatri Gopalan

Gene W Shen

Michael Nelson

R Hugo Patterson, Iii

Sean Lie

James Eugene Chow

Ming L So

Kevin Yikai Liang

Qi Gao

Dmitry Grinberg

Andrew Biggadike

John Cho

System and method for providing fault tolerance in virtualized computer systems use a first guest and a second guest running on virtualization software to produce outputs, which are produced when a workload is executed on the first and second guests. An output of the second guest is compared with an output of the first guest to determine if there is an output match. If there is no output match, the first guest is paused and a resynchronization of the second guest is executed to restore a checkpointed state of the first guest on the second guest. After the resynchronization of the second guest, the paused first guest is caused to resume operation.

Virtual machine fault tolerance

In a computer system running at least a first virtual machine (VM) and a second VM on virtualization software, a computer implemented method for the second VM to provide quasi-lockstep fault tolerance for the first VM includes executing a workload on the first VM and the second VM that involves producing at least one externally visible output and comparing an externally visible output of the second VM with an externally visible output of the first VM to determine if there is an output match. In response to a determination that the externally visible output of the second VM does not match the externally visible output of the first VM, a resynchronization of the second VM is executed. The externally visible output of the first VM is kept from being output externally until completion of the resynchronization.

The output of a non-deterministic instruction is handled during record and replay in a virtual machine. An output of a non-deterministic instruction is stored to a buffer during record mode and retrieved from a buffer during replay mode without exiting to the hypervisor. At least part of the contents of the buffer can be stored to a log when the buffer is full during record mode, and the buffer can be replenished from a log when the buffer is empty during replay mode.

Efficient recording and replaying of non-deterministic instructions in a virtual machine and CPU therefor

The translation lookaside buffer (TLB) of a processor is kept in synchronization with a guest page table by use of an indicator referred to as a 'T' bit. The T bit of the NPT/EPT entries mapping the guest page table are set when a page walk is performed on the NPT/EPT. When modifications are made to pages mapped by NPT/EPT entries with their T bit set, changes to the TLB are made so that the TLB remains in synchronization with the guest page table. Accordingly, record/replay of virtual machines of virtualized computer systems may be performed reliably with no non-determinism introduced by stale TLBs that fall out of synchronization with the guest page table.

Synchronizing a translation lookaside buffer with page tables

A distributed storage system has at least one host and at least one storage node. Hosts write data to and read data from storage devices within the storage nodes. A host upgrade manager (HUM) is installed in at least one of the hosts; a secondary upgrade manager (SUM) is installed in at least one of the nodes; and a primary upgrade manager (PUM) is installed within the distributed storage system itself, for example, in one of the nodes or hosts. Host and node storage system software is upgraded by transferring, via the respective HUM and SUM, upgraded versions of the software from the PUM to at least one host or node detected to be running an outdated version. The host and node storage system software components comprise different, cooperating portions of overall storage management software.

Coordinated upgrade of a cluster storage system

In a computer system running at least a first virtual machine (VM) and a second VM on virtualization software, a computer implemented method for the second VM to provide quasi-lockstep fault tolerance for the first VM. The method includes enabling operations of the second VM to diverge from being in lockstep with operations with the first VM without resynchronizing the second VM with the first VM, if output of the second VM is not diverging from output of the first VM. The output of the second VM is compared with the output of the first VM to determine whether the output of the second VM has diverged from the output of the first VM. In response to a determination that the output of the second VM has diverged from the output of the first VM, a resynchronization of the second VM is executed.

A virtualized computer system provides fault tolerant operation of a primary virtual machine. In one embodiment, this system includes a backup computer system that stores a snapshot of the primary virtual machine and a log file containing non-deterministic events occurring in the instruction stream of the primary virtual machine. The primary virtual machine periodically updates the snapshot and the log file. Upon a failure of the primary virtual machine, the backup computer can instantiate a failover backup virtual machine by consuming the stored snapshot and log file.

Low overhead fault tolerance through hybrid checkpointing and replay

A server system performs error detection on a hop-by-hop basis at multiple compute nodes, thereby facilitating the detection of a compute node experiencing failure. The server system communicates a packet from an originating node (the originating node) to a destination node by separating the packet into multiple flow control digits (flits) and routing the flits using a series of hops over a set of intermediate nodes. The packet's final flit includes error detection information, such as checksum data. As each intermediate node receives the final flit, it performs error detection using the error detection information. The pattern of nodes that detect an error indicates which intermediate node has experienced a failure.

Hop-by-hop error detection in a server system

A method and system for determining potential deadlock conditions in a target multi-threaded software application. The target application is first run in a virtual machine and the events within the application are recorded. The recorded events are replayed and analyzed to identify potential lock acquisition conflicts occurring between threads of the application. The potential lock acquisition conflicts are identified by analyzing the order in which resource locks are obtained and pairs of resources that have respective locks obtained in different orders are analyzed. These analyzed pairs are used to define a different order of events in the target application that, when the target application is re-run with the second order of events, may trigger a deadlock condition. The target application is then re-run with the different order of events in an attempt to trigger and then identify potential deadlock situations.

Detecting resource deadlocks in multi-threaded programs by controlling scheduling in replay

A method for precisely counting guest branch instructions in a virtualized computer system is described. In one embodiment, guest instructions execute in a direct execution mode of the virtualized computer system. The direct execution mode operates at a first privilege level having a lower privilege than a second privilege level. A branch count of previously executed first privilege level branch instructions is maintained as instructions execute. Execution of a first privilege level branch instruction caused by a control transfer to the direct execution mode is detected. Responsive to the detection, a guest branch instruction count is determined based on the first privilege level branch count.

Precise branch counting in virtualization systems

One embodiment is a computer-implemented method of trace collection for a virtual machine that includes: executing a sequence of instructions from an initial state of the virtual machine; accessing an event log of data relating to nondeterministic events, which data includes an execution point; making at least a portion of the data available to the virtual machine when the sequence reaches the execution point; collecting trace information in response to expansion parameters; and storing the trace information in a trace file.

Growing community of inventors

Palo Alto, CA, United States of America

Min Xu