Microsoft Technology Licensing, LLC

Microsoft Corportion

Scott Norin

Max L Benson

Darren Arthur Shakib

Dmitriy Meyerzon

Srikanth Shoroff

Dinarte R Morais

Milind Madhukar Joshi

Thomas F Fakes

F Soner Terek

William P Champion

A Web crawler application takes advantage of a document store's ability to provide a content identifier (CID) having a value that is a unique function of the physical storage location of a data object or document, such as a Web page. In operation, the crawler first tries to fetch the CID for a document. If the CID attribute is not supported by the document store, the crawler fetches the document, filters it to obtain a hash function, and commits the document to an index if the hash function is not present in a history table. If the CID is available from the document store, the CID is fetched from the document store. The crawler then determines whether the CID is present in the history table, which indicates whether an identical copy of the document in question has already been indexed under a different URL. If the CID is present, indicating that the document has already been indexed, the new URL is placed in the history file but the document itself is not retrieved from the document store, nor is it filtered again to obtain a CID. If the CID is not present in the history table, the full document is retrieved and indexed. The CID data structure is an extension of a known globally unique ID (GUID). Whereas the GUID is a 16-byte number, the CID comprises a 16-byte GUID plus an additional 6-byte number.

Method and system for detecting duplicate documents in web crawls

A system and method for incremental change synchronization among multiple copies of data is disclosed. To achieve the synchronization a genera synchronization model is used. The synchronization model establishes a one-way data flow path to allow incremental changes to be transferred from a copy of data that incorporates the incremental changes to a copy of data that does not yet incorporate the incremental changes. To achieve this one-way data flow path, the synchronization model uses an agent, a data collector, and a data synchronizer. The data collector collects incremental changes transferred to it and applies the incremental changes to a copy of data. A data synchronizer transfers incremental changes to a data collector. An agent controls the synchronization process by making a connection between a data collector and a data synchronizer and then directing the data synchronizer to transfer data to the data collector. If bidirectional data flow is necessary to achieve synchronization between two copies of data, a one-way data flow path is established first in one direction and after all incremental changes have been passed in that direction, a data flow path is established in the opposite direction to pass incremental changes in the opposite direction.

System and method for incremental change synchronization between multiple copies of data

Systems and methods for discovery based data recovery in a store and forward replication system are presented. Data loss is discovered by comparing a list of changes made to a local copy of a replica object with a list of changes received over the network from other nodes also having a copy of the replica object. When the list of changes received contains changes that the local list does not, the local system knows its copy of the replica object is not up-to-date. Missing changes are then requested from other systems having the missing data. In making the request, care is taken to minimize the cost incurred in recovering the missing data and to balance network traffic among several other nodes, if possible.

System and method for discovery based data recovery in a store and

A system and method for asynchronous store and forward data replication is presented. The system and method utilizes one-way, unacknowledged communication messages between systems in a computer network to transfer locally modified data to other systems having a copy of the data. The asynchronous store and forward data replication system and method utilizes existing computer networks and/or network control software as a transport agent to deliver the communication messages. Utilization of one-way, unacknowledged communication messages allows data replication across foreign networks such as the Internet or other wide area networks.

System and method for asynchronous store and forward data replication

A system and method for replicating hierarchical data is disclosed. The system and method preferably use one-way, unacknowledged communication messages to transfer data among various servers in a computer network. In many instances replicating hierarchically structured data requires processing the data in a hierarchical fashion even though the data is received in essentially random order. Hierarchically structured data is processed in the proper order by dynamically reconstructing the hierarchy as messages are received and processed. The invention first stores received replication packets in an incoming packet store. The data is processed by creating certain structures in memory for each corresponding replication packet and then processing all entries in the structures that can be processed. Global lists are kept for entries that remain unprocessed. If entries remain unprocessed because of hierarchical dependence on unprocessed data, the structures will remain in memory for a period of time in anticipation that the parent will soon be processed. When parent data is processed, the global lists are checked for child data that can then be processed.

System and method for the distribution of hierarchically structured data

Systems and methods of time based data expiration in a replication environment are presented. The systems and methods of the present invention allow different age limits to be set on each individual server in a network. A network can therefore be configured with some servers having no age limit and other servers having a wide range of differing age limits. Systems and methods for handling mixed data sets that include some data objects which are allowed to expire and some data objects which do not expire are also presented. In all cases, the time based data expiration systems and methods are designed to insure proper operation of the replication process and any associated data recovery process. Specifically, expired data will not be recovered via any data recovery mechanism and servers will not attempt to recover missing data from a server that has expired that data.

Time based expiration of data objects in a store and forward replication

A system and method for replica administration without data loss is disclosed. In a replication environment where data is replicated around a network and where any system can make changes to the data, data loss may occur if one copy of the data is deleted before changes made to that copy are replicated to other systems in the network. The present invention describes a robust administration environment which prevents inadvertent data loss by verifying that changes made to a local copy of the data reside on at least one other system in the network. The system and method of the present invention also provide a mechanism to allow an administrator to bypass such safeguards in appropriate circumstances in order to handle special cases such as total removal of the data from the network. The replica administration environment is implemented by defining various states that represent the level of participation in the replication of the data. For example, an active state can indicate full participation while a deleted state can indicate no participation. In addition to an active state and a deleted state, one or more intermediate states may be defined. The checks and safeguards can be performed in these intermediate states. In appropriate circumstances, an administrator may force the transition from certain of the intermediate states to either the active state in order to return a system to full participation or to the deleted state in order to bypass the safeguards of the present invention.

Replica administration without data loss in a store and forward

A system and method for distributed conflict resolution between different versions of the same data object which are replicated across a computer network is disclosed. The invention can be utilized with and adapted to any desired replication process. The conflict resolution process allows identification and resolution of conflicts with little or no increased message traffic on the network. The conflict resolution process is independent of the order in which conflicts are resolved and all replica nodes in the enterprise will eventually resolve the conflict in exactly the same manner. The method is adapted for resolution of conflicts between data objects or between objects which define the properties of sets of data objects. The final selection of which version of the object should be maintained and which version of the object should be deleted can be fully automated, completely manual, or a combination of the two.

System and method for distributed conflict resolution between data

A single processor/storage device provides multilingual support for a computer system. Information is divided into two parts. The first part is initially stored in a first character set, e.g., the character set in which it was created. A tag, identifying the character set, is associated with the first part. The second part is stored in a universal character set. The first part is converted to another character set only when requested. Indices may be generated from the second part using specified language sorting rules.

Growing community of inventors

Newcastle, WA, United States of America

Scott Norin