Google Inc.

Hewlett-packard Development Company, L.P.

Keith H Randall

Sanjay Ghemawat

Michael Burrows

Monika Hildegard Henzinger

Anurag Acharya

Robert J Stets, Jr

Daniel Wesley Dulitz

Raymond Paul Stata

Zhe Qian

Arash Baratloo

Bwolen Po-Jen Yang

Wilburt Labio

Ming-Yee Iu

Rajiv G Wickremesinghe

Systems and methods for scheduling document crawling are provided in which a list of document identifiers is obtained. Each respective document identifier identifies a corresponding document on a network. For each respective document identifier in the list of document identifiers, a content change frequency of the corresponding document is determined and a first score for the document identifier that is a function of the determined content change frequency of the corresponding document is also determined. The first score is compared against a threshold value. The document is scheduled for crawling based on the result of the comparison. The content change frequency for a respective document identifier is determined by comparing information stored for successive downloads of the document corresponding to the document identifier.

Scheduler for search engine crawler

The present invention relates generally to a system and method for measuring application memory use, and more particularly to measuring heap usage of each of a plurality of applications running inside a single heap. Preferred embodiments of the present invention work by traversing a set of objects in a heap. During this traversal, sets of strongly connected components are identified. Additionally, representative objects of the sets of strongly connected components are identified and a topological sort order of the objects is established. Further, during a second traversal of the objects, the topological sort order is used to identify one or more applications responsible for each of the strongly connected component sets. And, in the process, the resource usage of each application is computed.

System and method of measuring application resource usage

An advertiser may be provided with fast and accurate estimates of the future cost and/or performance information for one or more actual or hypothetical ads (generally referred to as 'proto-ads'). Past auction information may be used to simulate auctions that the proto-ad would have competed in. The proto-ad may then participate in a 'replay' of such past auctions.

Estimating cost and/or performance information for an advertisement in an advertising system

A scheduler for a search engine crawler includes a history log containing document identifiers (e.g., URLs) corresponding to documents (e.g., web pages) on a network (e.g., Internet). The scheduler is configured to process each document identifier in a set of the document identifiers by determining a content change frequency of the document corresponding to the document identifier, determining a first score for the document identifier that is a function of the determined content change frequency of the corresponding document, comparing the first score against a threshold value, and scheduling the corresponding document for indexing based on the results of the comparison.

A method of distributing files operates in a system having a master and a plurality of slaves, interconnected by a communications network. Each slave determines a current file length for each of a plurality of files and sends slave status information to the master, the slave status information including the current file length for each file. The master schedules copy operations based on the slave status information. The master stores bandwidth capability information indicating data transmission bandwidth capabilities for the resources required to transmit data between the slaves, and also stores bandwidth usage information indicating a total allocated bandwidth for each resource. For each schedule copy operation, an amount of data transmission bandwidth is allocated and the stored bandwidth usage information is updated accordingly. The master only schedules copy operations that do not cause the total allocated bandwidth of any resource to exceed the bandwidth capability of that resource.

System and method for data distribution

A web crawler system includes a central processing unit for performing computations in accordance with stored procedures and a network interface for accessing remotely located computers via a network. A web crawler module downloads pages from remotely located servers via the network interface. A first link processing module obtains page link information from the downloaded page; the page link information includes for each downloaded page a row of page identifiers of other pages. A second link processing module encodes the rows of page identifies in a space efficient manner. It arranges the rows of page identifiers in a particular order. For each respective row it identifies a prior row, if any, that best matches the respective row in accordance with predefined row match criteria, determines a set of deletes representing page identifiers in the identified prior row not in the respective row, and determines a set of adds representing page identifiers in the respective row not in the identifier prior row. The second link processing module delta encodes the set of deletes and delta encodes the set of adds for each respective row, and then Huffman codes the delta encoded set of deletes and delta encoded set of adds for each respective row.

Growing community of inventors

Mountain View, CA, United States of America

Keith H Randall