Google Inc.

Howard Bradley Gobioff

Sanjay Ghemawat

Shun-Tak Leung

Anurag Acharya

Huican Zhu

Maximilian Ibel

Max Ibel

David L desJARDINS

A system facilitates the distribution and redistribution of chunks of data among multiple servers. The system may identify servers to store a replica of the data based on at least one of utilization of the servers, prior data distribution involving the servers, and failure correlation properties associated with the servers, and place the replicas of the data at the identified servers. The system may also monitor total numbers of replicas of the chunks available in the system, identify chunks that have a total number of replicas below one or more chunk thresholds, assign priorities to the identified chunks, and re-replicate the identified chunks based substantially on the assigned priorities. The system may further monitor utilization of the servers, determine whether to redistribute any of the replicas, select one or more of the replicas to redistribute based on the utilization of the servers, select one or more of the servers to which to move the one or more replicas, and move the one or more replicas to the selected one or more servers.

Systems and methods for replicating data

Systems and method are provided for setting a respective reuse flag for a corresponding document in a plurality of documents based on a query-independent score associated with the corresponding document. A document crawling operation is performed on the plurality of documents in accordance with the reuse flag for respective documents in the plurality of documents. This document crawling operation includes reusing a previously downloaded version of a respective document in the plurality of documents instead of downloading a current version of the respective document from a host computer in accordance with a determination that the reuse flag associated with the respective document meets a predefined criterion.

Document reuse in a search engine crawler

A search engine crawler includes a scheduler for determining which documents to download from their respective host servers. Some documents, known to be stable based on one or more record from prior crawls, are reused from a document repository. A reuse flag is set in a scheduler record that also contains a document identifier, the reuse flag indicating whether the document should be retrieved from a first database, such as the World Wide Web, or a second database, such as a document repository. A set of such scheduler records are used during a crawl by the search engine crawler to determine which database to use when retrieving the documents identified in the scheduler records.

A search engine crawler includes a distributed set of schedulers that are associated with one or more segments of document identifiers (e.g., URLs) corresponding to documents on a network (e.g., WWW). Each scheduler handles the scheduling of document identifiers (for crawling) for a subset of the known document identifiers. Using a starting set of document identifiers, such as the document identifiers crawled (or scheduled for crawling) during the most recent completed crawl, the scheduler removes from the starting set those document identifiers that have been unreachable in each of the last X crawls. Other filtering mechanisms may also be used to filter out some of the document identifiers in the starting set. The resulting list of document identifiers is written to a scheduled output file for use in a next crawl cycle.

Scheduler for search engine crawler

A system that facilitates the distribution and redistribution of chunks of data among multiple servers, may identify servers to store replicas of the chunks based on at least one of utilization, prior data distribution, and failure correlation properties, and place the replicas at the identified servers. The system may monitor total numbers of replicas available in the system, identify chunks that have a total number of replicas below one or more thresholds, assign priorities to the identified chunks, and re-replicate the identified chunks based on the assigned priorities. The system may monitor utilization of the servers, select one or more of the replicas to redistribute based on the utilization of the servers, select one or more of the servers to which to move the one or more replicas, and move the one or more replicas to the selected one or more servers.

A system facilitates the deletion of data, such as files, orphaned chunks, and stale replicas. The system may identify a file to be deleted, rename the identified file, permanently delete the renamed file a predetermined amount of time after renaming the identified file as part of a garbage collection process, receive, from the servers, information concerning chunks stored by the servers, and identify, to the servers, ones of the chunks that do not exist possibly due to the permanent deletion of the renamed file. The system may further provide a mapping of file names to chunks, identify chunks, as orphaned chunks, that are not reachable from any of the file names, delete the orphaned chunks, receive, from the servers, information concerning chunks stored by the servers, and identify, to the servers, ones of the chunks that are orphaned chunks. The system may also associate version information with replicas of chunks, identify stale replicas based on the associated version information, delete the stale replicas, receive, from the servers, information concerning replicas stored by the servers, and identify, to the servers, ones of the replicas that are stale replicas.

Garbage collecting systems and methods

A file system includes a master connected to multiple servers. The servers may store data. The master may communicate with the servers upon startup of the master to authoritatively identify the data stored by the servers and record location information that identifies ones of the servers that store the data.

Maintaining data in a file system

A system may perform a first operation within a file system in which directories and files are organized as nodes in a namespace tree. The system may associate a read-write lock with each of the nodes in the namespace tree. The system may acquire a first lock on a name of one or more directories involved in the first operation, acquire a second lock on an entire pathname involved in the first operation, determine whether the first lock or the second lock conflicts with third locks acquired by a second operation, and perform the first operation when the first lock or the second lock does not conflict with the third locks. The first, second, and third locks may include read-write locks.

Growing community of inventors

San Francisco, CA, United States of America

Howard Bradley Gobioff