Sun Microsystems, Inc.

Madhusudhan Talluri

Yousef A Khalidi

Declan J Murphy

Andrew G Tucker

Marshall C Pease

Charles E Narad

Srinivasan Viswanathan

Stephen Alan Chessin

Jose M Bernabeu-Auban

Shing I Kong

Skef F Iterum

Anil Swaroop

Ying Xie

David Dion

Glen R Anderson

Vikram P Joshi

Michael Lee Federwisch

Vladimir Matena

Kenneth William Shirriff

James J Voll

Bart Smaalders

Kevin C Fox

Timothy Paul Marsland

Sundeep R Panicker

Steven K Fought

Dock G Williams

Krishna Kondaka

Jongki A L Suwandi

Method for enabling the dynamic modification of cluster configurations, and apparatus including software to perform the method. To enable this dynamic modification, cluster configuration data is stored as a table in a cluster configuration repository that is accessible from all nodes in the cluster. Accordingly, the present invention enables the modification of the cluster configuration from any node in the cluster dynamically. When a reconfiguration command is given, the configuration table is changed and all the nodes in the cluster are notified of the changed configuration in parallel. Following the notification by the nodes of the changed cluster configuration, the changes to the cluster are implemented dynamically as specified by the command.

Method to dynamically change cluster or distributed system configuration

One embodiment of the present invention provides a system for testing a computer system by using software to inject faults into the computer system while the computer system is operating. This system operates by allowing a programmer to include a fault point into source code for a program. This fault point causes a fault to occur if a trigger associated with the fault point is set and if an execution path of the program passes through the fault point. The system allows this source code to be compiled into executable code. Next, the system allows the computer system to be tested. This testing involves setting the trigger for the fault point, and then executing the executable code, so that the fault occurs if the execution path passes through the fault point. This testing also involves examining the result of the execution. In one embodiment of the present invention, if the fault point is encountered while executing the executable code, the system executes the fault point by: looking up a trigger associated with the fault point; determining whether the trigger has been set; and executing code associated with the fault point if the trigger has been set.

Method and apparatus for testing a computer system through software fault injection

A system and method are provided for ensuring delivery of a communication from one computer system or node to another. A first node includes an object handler, such as an ORB (Object Request Broker), that receives object references from higher-level services operating on the first node, wherein the referenced object resides on a second node. The first node's object handler generates a message to an object handler on the second node and attempts to send the message to the second node through a transport module. The message is assigned a unique identifier, such as a sequence number. If the first object handler receives an uncertain status concerning the message (e.g., other than a certain success or failure), it issues a query to the second node to determine if the message was received. If the query is received by the second object handler before the message itself is received, the message is considered lost or rescinded by the first node. The first node stores the identifier so that it will not be re-assigned to another message and the message is then re-sent with a different identifier. The second object handler notes the identifier and status of the rescinded message and will discard any message having that identifier that is received. The second node includes two or more data structures to track the status of communications sent from the first node. The first node, in addition to a collection of identifiers of lost messages, may also record the status of communications it attempts to send and may also note the identifiers of messages that could not be transmitted (e.g., because of communication link errors).

System and method for ensuring delivery of a single communication between nodes

A first computer sends a sequence of messages to a second computer using remote write operations to directly store each message in a corresponding memory location in the second computer. The second computer retains information denoting the sequence numbers of the messages it receives and processes, and it acknowledges each received message with an asynchronous acknowledgment message. The first computer keeps track of which messages it has sent but for which it has not yet received an acknowledgment. Whenever the first computer determines that it has failed to receive a message acknowledgment from the second computer in a timely fashion, or it needs to reuse previously used message sequence numbers, the first computer undertakes remedial actions to resynchronize the first and second computers. The process begins by prompting the second computer to flush and process all the messages in its receive FIFO, and then comparing sequence number information recorded by the second computer with the sequence numbers of the outstanding, unacknowledged messages sent by the first computer. If the comparison indicates that any messages sent by the first computer were not received and processed by the second computer, those messages are re-transmitted. If necessary, during resynchronization the first computer will activate a different communication interface than the one previously used so as to establish a reliable connection to the second computer. After a success resynchronization, normal &ldquo;send only&rdquo; message operation resumes. At predefined times, the sequence number information retained by the second computer is cleared.

System and method for message transmission between network nodes connected by parallel links

One embodiment of the present invention provides a system that facilitates communications between a cluster of nodes within a clustered computing system in a manner that tolerates failures of communication pathways between the nodes. The system operates by configuring a distinct logical pathway between each possible source node and each possible destination node in the cluster, so that each distinct logical pathway is routed across one of at least two disjoint physical pathways between each possible source node and each possible destination node. In doing so, the system configures a first logical pathway between a first node and a second node across a first physical pathway of at least two disjoint physical pathways between the first node and the second node. Upon detecting a failure of the first physical pathway, the system reroutes the first logical pathway across a second physical pathway from the at least two disjoint physical pathways between the first node and the second node. In one embodiment of the present invention, the system associates a distinct per-node logical address with each node in the cluster. For each source node, the system associates the per-node logical address of each possible destination node with a corresponding logical pathway to the destination node. In this way, a communication from a given source node to a per-node logical address of a given destination node is directed across the corresponding logical pathway to the given destination node.

Method and apparatus for providing fault-tolerant addresses for nodes in a clustered system

One embodiment of the present invention provides a method and an apparatus that facilitates transparent failovers from a primary copy of an object on a first server to a secondary copy of the object on a second server when the first server fails, or otherwise becomes unresponsive. The method includes detecting the failure of the first server; selecting the second server; and reconfiguring the second server to act as a new primary server for the object. Additionally, the method includes transparently retrying uncompleted invocations to the object to the second server, without requiring explicit retry commands from a client application program. A variation on this embodiment further includes winding up active invocations to the object before reconfiguring the second server to act as the new primary server. This winding up process may include causing invocations to unresponsive nodes to unblock and complete. Another variation includes blocking new invocations to the object after detecting the failure of the first server, and unblocking these new invocations after reconfiguring the second server to act as the new primary server. Hence, the present invention can greatly simplify programming of client application programs for highly available systems. It also makes it possible to use a client application program written for a nonhighly available system in a highly available system.

Method and apparatus for transparent server failover for highly available objects

A file disaster recovery system that employs geographical replication of data from a local site to remote site in a such a manner that file requests from clients of the local site can be handled by a file server on the remote site following a failover from the local site to the remote site. Geographical data replication software running on a local server checkpoints to a log in local stable storage all information on file operations that change the file state of the local file system. According to a selected mode, the local geographical data replication software flushes information in the log pertaining to the file operations since the last flush to the remote site. At the remote site, compatible remote geographical data replication software running on a remote file server receives the flushed log and replicates in sequence order the file operations represented in the flushed log. The results of the operations are stored on remote stable storage. The local and remote servers can be clusters or single servers. There is no need for commonality, except for operating and file systems, between the local and remote sites. Because operations are replicated and not low level, formatted data, the local and remote stable file storage can be completely different.

Method and apparatus for file system disaster recovery

The present invention pertains to a system and method for performing remote object invocation. A object-oriented computing system includes a number of independent computing nodes that are interconnected via a communications link. The nodes represent client and/or server computers that do not share memory. Each node includes a number of domains having separate address spaces. Each domain includes one or more threads of execution that invoke one or more objects. The object's method can reside in the same domain as the requesting thread, in a different domain within the same node, or in a different domain in another node. A file descriptor is used to represent those objects whose methods reside in a different domain than the requesting thread. A file descriptor is a protected kernel entity that enables a thread to invoke an object. A thread can only access those objects for which it has received an associated file descriptor. An Object Request Broker (ORB) services the object invocation requests for those objects residing in different domains. The ORB provides a mechanism that translates the file descriptor used to reference an object in one domain into the file descriptor used to invoke the object in the domain having the object's method.

System and method for remote object invocation

The present invention pertains to a system and method for tracking object references in an object-oriented computing system including a number of independent computing nodes interconnected by a communications link. The reference counting mechanism tracks references to an object through the use of a messaging protocol. A server node keeps a foreign reference count for each of its objects. The foreign reference count indicates the number of remote nodes having a reference to one of the servers objects. A server node increments the foreign reference count for each object reference that it exports prior to sending it to an intended client node. A client node will send a message to the server node when it has already received the exported object reference and in response to this message, the server node decrements the appropriate foreign reference count. When a client node receives an object reference from another client node, the receiving client node will message the appropriate server node that the receiving client node has a reference to the object. In response, the server node increments the appropriate foreign reference count. The reference counting mechanism utilizes a node failure protocol in order to accurately track the foreign reference counts in light of node failures.

Method and system for reliable remote object reference management

The present invention pertains to a system and method for simulating multiple clusters of independent computer nodes in a single machine. A cluster contains one or more computer nodes interconnected by a communications link. A user can simulate a cluster of n nodes by generating n user-level procedures where each user-level procedure represents the kernel of a respective node. An additional mechanism is provided which allows a user to exercise the operation of any intended function in any of the nodes in any of the clusters.

Apparatus and method for simulating multiple nodes on a single machine

When a client computer requests data from a disk or similar device at a server computer, the client exports the memory associated with an allocated read buffer by generating and storing one or more incoming MMU (IMMU) entries that map the read buffer to an assigned global address range. The remote data read request, along with the assigned global address range is communicated to the server node. At the server, the request is serviced by performing a memory import operation, in which one or more outgoing MMU (OMMU) entries are generated and stored for mapping the global address range specified in the read request to a corresponding range of local physical addresses. The mapped local physical addresses in the server are not locations in the server's memory. The server then performs a DMA operation for directly transferring the data specified in the request message from the disk to the mapped local physical addresses. The DMA operation transmits the specified data to the server's network interface, at which the mapped local physical addresses to which the data is transferred are converted into the corresponding global addresses. The specified data with the corresponding global addresses are then transmitted to the client node. The client converts the global addresses in the received specified data into the local physical addresses corresponding to the allocated receive buffer, and stores the received specified data in the allocated receive buffer.

System and method for efficient remote disk I/O

A first computer sends a sequence of messages to a second computer using remote write operations to directly store each message in a corresponding memory location in the second computer, without performing remote read operations to confirm storage of each message in the second computer's memory. The first computer detects message transmission errors, if any, during the transmission of each message to the second computer and when a message transmission error is detected, resends the message to the second computer. The first computer also sends trigger messages to the second computer to prompt the second computer to process messages stored in the second computer's memory. The second computer processes each received message and uses a remote write operation to store an acknowledgment message in a corresponding memory location in the first computer. The first computer, upon detecting a failure to receive the acknowledgment message corresponding to any of the previously sent messages, undertakes remedial actions to determine whether the second computer has processed each of the unacknowledged previously sent messages. When the remedial actions determine that the second computer has not processed the unacknowledged previously sent message, the first computer prompts the second computer to process the unacknowledged previously sent message. When the remedial actions determine that the second computer has already processed the unacknowledged previously sent message, the first computer stores the acknowledgment message in a corresponding location in the first computer's memory.

System and method for message transmission between network nodes using

In a distributed computer system, a sending node prompts a receiving node to allocate and export to the sending node one or more memory 'segments'. Each allocated segment is sufficiently large to hold multiple receive buffers whose size fall within a predefined range of receive buffer sizes. Once a segment has been allocated and exported, the sending node allocates receive buffers within the segment, using sequentially contiguous portions for successive receive buffers, without any interaction with the receiving node. Messages are transmitted to the receiving node by remotely writing the data portion of each message to an allocated receive buffer and writing a control message with a pointer to the corresponding receive buffer to a message queue in the receiving node. The receiving node processes messages within the portions of the allocated segments specified by each control message and does not keep track of the used and/or unused portions of each segment. Only the sending node keeps track of the unused portions of each allocated segment. As a result, the computational and communication resource overhead associated with allocating receive buffers is substantially reduced. Once the sending node has exhausted a segment, it sends either a segment release message or segment recycle message to the receiving node. Recycling the segment causes the receiving node to re-enable the sending node to use the segment once the receiving node has processed all the messages previously written to receive buffers in the segment.

System and method for remote buffer allocation in exported memory

A method and apparatus for translating a virtual address to a physical address. A virtual address to be translated has a virtual page offset and a virtual page number. The virtual address to be translated addresses a page of memory. The size of this page is unknown. There are L different possible page sizes where L is a positive integer greater than one. Each of the L different page sizes is selected to be a test page size and a test is performed. During the test, a pointer into a translation storage buffer is calculated. The pointer is calculated from the virtual address to be translated by assuming that the virtual address to be translated corresponds to a mapping of the test page size. The pointer points to a candidate translation table entry of the translation storage buffer. The candidate translation table entry has a candidate tag and candidate data. The candidate tag identifies a particular virtual address and the candidate data identifies a particular physical address corresponding to the particular virtual address. A virtual address target tag is extracted from the virtual address to be translated. The virtual address target tag is calculated by assuming that the virtual address to be translated corresponds to a mapping of the test page size. The target tag and the candidate tag are then compared. If the target tag matches the candidate tag, the candidate data is provided as the physical address translation corresponding to the virtual address to be translated.

Growing community of inventors

Fremont, CA, United States of America

Madhusudhan Talluri