Microsoft Technology Licensing, LLC

Compaq Computer Corporation, Inc.

Digital Equipment Corporation

Leslie B Lamport

Michael T Massa

James Michael Reuter

Eliezer Gafni

John R Douceur

Daniel J Scales

Aamer Hydrie

Jacob Rubin Lorch

Jonathan Ryan Howell

Dahlia Malkhi

Lidong Zhou

Thomas Lee Rodeheffer

Demetrios Achlioptas

K Mani Chandy

A distributed computing system can operate in the face of malicious failures on the part of some of its constituent devices, and provide a minimum of message delays between receiving a client request and providing a response, when each device within the system verifies the sender of any message it receives, and the propriety of the message.

Selecting values in a distributed computing system

Synchronized devices comprising a distributed system attempt to agree on a compatible sequence of commands to execute. Each device in the distributed system may act as a proposer, acceptor, or a learner. Each proposer proposes a command for each device to execute. The acceptors either accept or reject the proposed commands. The learners keep track of the proposed commands and determine, using a transactional substrate, whether the acceptors have a accepted sequences of commands that commute with respect to one another. Once the learners have determined that a quorum of acceptors have accepted sequences of commands that commute with respect to one another the accepted commands are executed by each device in the distributed system.

Automatic commutativity detection for generalized paxos

A conflict tolerant message delay reducing consensus algorithm is presented for operating a distributed computing system. The devices of the distributed computing system can directly receive client requests, and can execute the requests and respond directly to the clients, saving message delays. If there is a conflict, the ultimately selected request can be the request submitted by the client with the highest client identifier. A device can change its vote, and execute a different request, if it is made by a client having a more dominant client identifier. All but one of the clients can also be a device implementing the system. A device that has executed a requested function may no longer submit a request in the same step. Consequently, a request is executed by the system when all devices have executed the request. If one or more devices fails, any fault tolerant consensus algorithm can be used.

Conflict fast consensus

A distributed computing system can be operated in a fault tolerant manner using a collection of auxiliary computing devices and more main computing devices than the number of faults the system can tolerate. A quorum of all of the main computing devices can be used. In the event of a failure, an alternative quorum from a selected set of quorums, comprising at least one main computing device and some or all of the auxiliary computing devices, can be used to complete pending operations and to select a new set of quorums. Alternatively, another state machine, comprising at least one main computing device and some or all of the auxiliary computing devices, can select a new quorum comprising the currently operating main computing devices, and the new quorum can then complete pending operations and can continue to select proposals using the proposal number assigned by the other state machine.

Cheap paxos

A variant of Paxos is referred to as Virtually Synchronous Paxos (VS Paxos). VS Paxos is a self-reconfigurable protocol that allows for delay only for reconfiguration decisions, without placing an artificial limit on regular decisions. In an implementation of VS Paxos, subject to any restriction on reconfiguration decisions, a leader may activate an unbounded number of consensus instances ahead. A VS Paxos technique permits unlimited progress during stability periods, in that a leader may initiate commands at any number of consensus instances without bound. VS Paxos waits for command completion only when configuration-changing commands occur.

Virtually synchronous Paxos

The leader server proposing commands or sequences of commands to be executed by a plurality of servers according to an asynchronous consensus algorithm, such as Paxos, is replaced with a synchronous virtual leader agreement algorithm executed by each of the servers. The synchronous agreement algorithm used to determine the virtual leader's actions is called the leader agreement algorithm. The asynchronous consensus agreement algorithm performed by the virtual leader and the servers may be the Paxos algorithm. However, any asynchronous Byzantine agreement algorithm may be used. By replacing the leader in the Paxos consensus algorithm with a synchronous virtual leader agreement algorithm, the Paxos algorithm can survive a Byzantine failure of one or more leader server, as long as the servers are behaving synchronously.

Leaderless byzantine consensus

A simplified fault tolerant algorithm is presented for operating a distributed computing system in a fault tolerant manner. A system comprising three computing devices need only have two devices agree to perform any proposed function. Thus, when soliciting a vote for a proposed function, a leader device can also send its vote for the proposed function. This allows any recipient device to complete the quorum with its own vote. Consequently, any recipient device can, without any further messages, determine whether to execute the proposed function. Furthermore, if the device executes the proposed function, it can transmit the results directly to the client that requested the function, saving a message delay. If the quorum of devices used to select and execute proposed functions is itself selected by a quorum, then one of the devices of the system can be an inexpensive device having limited computational ability or storage capacity.

Simplified Paxos

A distributed computing system can achieve a generalized consensus, enabling commands that commute to be selected in any order. A leader can learn of previously selected sequences of commands, and can propose a compatible sequence of commands. Devices can select a sequence of commands that is compatible with previously selected sequences. Additional commands can be selected by selecting a sequence of commands comprising a previously selected sequence and the additional commands. Further efficiencies can be realized if the devices receive proposals directly from clients. Two or more proposals arriving in varying orders at the various clients may be selected in varying orders. However, if those proposals commute, a generalized consensus nevertheless exists despite the variations, enabling the system to continue efficient operation. To conserve memory, a checkpoint command that does not commute with any other command can be used to secure a sequence of commands for archiving or deletion.

Generalized Paxos

A distributed computing system can operate in the face of malicious failures on the part of some of its constituent devices, and provide a minimum of message delays between receiving a client request and providing a response, when each device within the system verifies the sender of any message it receives, and the propriety of the message. The sender can be verified through message authentication schemes or digital signature schemes. The propriety of a message can be verified by receiving a sufficiently large number of equivalent, properly authenticated messages. If the number of malicious devices is represented by the variable 'M', a sufficient number of equivalent, properly authenticated messages to verify that the message is true can be any number of messages greater than M. Furthermore, to verify that a leader device is not maliciously submitting different proposals to different devices using the same proposal number, a quorum of devices can be required to select a proposal, where a quorum is a sufficiently large number of devices such that any other quorum has, as a majority of its devices, non-malicious devices from the first quorum. Therefore, the distributed computing system can operate properly with M number of malicious failures and F number of total failures, and with a minimum of message delays, if the number of constituent devices in the distributed computing system is greater than 3F+2M. Additionally, if the distributed computing system can revert to a more traditional algorithm if too many devices fail or become malicious, it can use a message-delay-reducing algorithm while having as few as 2Q+F+2M+1 constituent devices, where Q is the number of devices that can fail and still allow the system to use a message-delay-reducing algorithm.

Fast byzantine paxos

A distributed computing system can operate in the face of malicious failures on the part of some of its constituent devices when each device within the system verifies the sender of any message it receives, and the propriety of the message. The sender can be verified through message authentication schemes or digital signature schemes, though message authentication can provide a more computationally efficient solution. The propriety of a message can be verified by receiving a sufficiently large number of equivalent, properly authenticated messages such that, even if every malicious device transmitted a message, at least one message would have been sent by a properly functioning device. If the number of malicious devices is represented by the variable 'M', a sufficient number of equivalent, properly authenticated messages to verify that the message is true can be any number of messages greater than M. Furthermore, the receipt of more than 2M equivalent properly authenticated messages can allow the receiving device to prove the propriety of the message to any device it forwards the messages onto. The proper operation of the distributed computing system can, therefore, proceed in the face of M number of malicious failures and F number of total failures, which can include malicious and non-malicious failures, if the number of constituent devices in the distributed computing system is greater than 2F+M.

Byzantine paxos

A distributed computing system, having a sufficient number of devices and requiring a sufficiently large number of devices to select any proposal, can maintain synchronization between its constituent devices and respond to client requests with as few as two message delays. A leader can synchronize the devices of the system and establish a safe proposal number for all current and future steps of the system. Devices can then receive client requests directly, treating the request as a proposal having the proposal number determined previously by the leader, and voting for the proposal. If the client receives an indication from a least a quorum of devices, where a quorum can be the minimum number of devices that can be operational at a given time, the client can know that the request was selected. If two or more clients attempt to request different functions at approximately the same time, the system may not select either request, in which case a leader can be requested to determine if any requests may have been selected, and to reestablish a safe proposal number. Systems with fewer devices can also implement the message-delay-reducing algorithm if they can also revert to the standard Paxos algorithm if an insufficient number of devices are operational. Such algorithms can be used to provide an efficient method for determining whether to commit or abort a client transaction.

Fast transaction commit

A distributed computing system can achieve consensus while introducing fewer message delays by using an algorithm that allows the constituent devices to vote on functions received directly from one or more clients. If a conflict occurs, a leader device from among the devices can be selected such that the leader device already knows of the other devices' previous votes, and can determine an appropriate function to propose, using an immediately subsequent proposal number, without performing the first phase of the Paxos algorithm. Alternatively, each device can independently determine, by using the same repeatable mechanism used by a leader device, what function the leader device would propose, and can then vote for that function using the immediately subsequent proposal number. If the devices' votes again result in a conflict, the Paxos algorithm can be used, or additional iterations can be performed prior to resorting to the Paxos algorithm.

Fast Paxos recovery

New information is introduced to a distributed system at many places. The information impacts data that is replicated throughout the system. The replicas of the data must be synchronized so that changes to the system are accurately reflected. One solution to the synchronized replica problem is a state machine approach. In such an approach, the computers of the network each maintain their own data, which is kept in the same state as the data of the other computers by processing the identical commands in the identical order. To ensure that all commands are executed in the same order, a lead process is given the task of assigning command sequence numbers. However, using a single lead process introduces a bottleneck in the distributed system by requiring that all commands to change data go through the lead process to be assigned an execution order. The invention divides the command ordering among multiple lead processes. Each lead process assigns command sequence numbers that are ordered with respect to other lead process but which do not conflict with the command sequence numbers assigned by the other leader processes.

Multi-leader distributed system

A system and method for effectuating distributed consensus among members of a processor set in a multiprocessor computing system that is effective even when only a single surviving processor is operational and is achieved through joint implementation of a virtual state machine utilizing a sequence of numbered input commands. System synchronization is achieved by having all of the processors agree on the sequence of input commands so that they execute the same virtual state machine. Input commands are numbered consecutively and the processors use a set of shared stores (e.g. disk drives) to communicate amongst themselves requests (i.e. ballots) for new state machine inputs (or commands) and state machine inputs that have already been chosen (i.e. committed commands). A consensus process is used to decide upon (or commit) each command and this consensus is achieved using a majority of known system stores rather than a majority of known processors. Consequently, when consensus is achieved, it then exists on the system stores (e.g. the disk drives) and not in the processors.

System and method for effectuating distributed consensus among members of a processor set in a multiprocessor computing system through the use of shared storage resources

Growing community of inventors

Palo Alto, CA, United States of America

Leslie B Lamport