Google Inc.

Whowhere, Inc.

Quote LLC

Rupesh Kapoor

Erik Arjan Hendriks

David Michael Proudfoot

Joachim Kupke

Pawel Aleksander Fedorynski

Hui Xu

Hao Fang

Andrei Pascovici

Cristian Tapus

Sathayanarayana Giridhar

Ziga Mahkovec

Implementations include a batch-optimized render and fetch architecture. An example method performed by the architecture includes receiving a request from a batch process to render a web page and initializing a virtual clock and a task list for rendering the web page. The virtual clock stands still when a request for an embedded item is outstanding and when a task is ready to run. The method may also include generating a rendering result for the web page when the virtual clock matches a run time for a stop task in the task list, and providing the rendering result to the batch process. Another example method includes receiving a request from a batch process to render a web page, identifying an embedded item in the web page, and determining, based on a rewrite rule, that the embedded item has content that is duplicative of content for a previously fetched embedded item.

Batch-optimized render and fetch architecture

Batch-optimized render and fetch architecture utilizing a virtual clock

A system may identify a set of first documents associated with an organization, and identify clusters to which the first documents belong. Each of a number of the identified clusters may include a group of documents that includes one of the first documents and one or more second documents associated with one or more different organizations. The system may determine a quality score for each of the documents in each of the identified clusters, and determine, for each of the number of the identified clusters, whether the quality score of the one of the first documents in the identified cluster is higher than the quality score of the one or more second documents in the identified cluster. The system may generate a proxy pad score based on the determinations, and store the proxy pad score.

Detection of proxy pad sites

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating modified web documents. In one aspect, a method includes retrieving a web document including markup language code and having references to resources external to the web document and parsing the retrieved web document to interpret the markup language code and identify references to resources external to the retrieved web document. Data relating to at least a portion of the resources external to the retrieved web document are retrieved, and a modified web document including the retrieved data is generated and stored for use in responding to a request for retrieval of content of the web document.

Modifying web pages to reduce retrieval latency

System and method for indexing rendered web page images. A web crawling engine stores the content and crawl time of a web page. A scheduling engine sends the content and crawl time to a rendering engine, and processes requests for embedded objects. If a requested object has been crawled, it sends the contents to the rendering engine. Otherwise it schedules the crawl of the object, and once the object is crawled, it resends the content and crawl time of the web page to the rendering engine. The rendering engine receives the content and crawl time of a web page, requests all embedded objects, and renders the web page to an image once all embedded objects are received.

Iterative off-line rendering process

A system may identify a set of related documents, identify one or more documents in the set of related documents that are sources of redirects, and identify organizations that are targets of the redirects. The system may also determine a redirect score based on the number of the identified documents that are sources of the redirects, determine a spam score based on a number of the organizations that are targets of the redirects, determine whether to classify the set of related documents as a bounce pad based on the redirect score and the spam score, and storing information associated with the result of the determination of whether to classify the set of related documents as a bounce pad.

Detection of bounce pad sites

A URL inspector may determine a uniform resource locator (URL) which includes an indexable fragment. A URL separator may separate, from the URL which includes the indexable fragment, a base URL occurring prior to the indexable fragment. An indexer may process content of the base URL to obtain processed content thereof. A rendering system may render the processed content together with the URL which includes the indexable fragment to obtain rendered content. A content converter may convert the rendered content into indexable content.

Indexing of URLs with fragments

A method for providing Internet protocol (IP) addresses with a domain name server (DNS) for multiple web servers of an Internet host. In one embodiment, each web server of an Internet host having multiple web servers is assigned a relative weight based on the individual processing power of the particular web server. As DNS resolution requests are received from client domains, the DNS returns IP addresses for the web servers such that the total number of times that each IP address of the web server is returned is proportional to the relative weight of each server relative to the total weight of all the servers. In another embodiment, the client domains that have most frequently accessed the web servers according to recent web server access logs are identified. In addition, the total number of accesses of each client domain is considered such that static arrays may be constructed to apportion the web servers among the client domains that most frequently access the web servers. As such, each of the client domains that most frequently access the web servers are assigned to a particular web server such that the percentage of requests served by each particular web server is proportional to the relative weight of that web server. In doing so, the probability of a load imbalance between the web servers is reduced.

Growing community of inventors

Palo Alto, CA, United States of America

Rupesh Kapoor