Google Inc.

Pawel Aleksander Fedorynski

Rupesh Kapoor

Andrei Pascovici

Erik Arjan Hendriks

Sathayanarayana Giridhar

Hui Xu

Sumitro Samaddar

System and method for indexing rendered web page images. A web crawling engine stores the content and crawl time of a web page. A scheduling engine sends the content and crawl time to a rendering engine, and processes requests for embedded objects. If a requested object has been crawled, it sends the contents to the rendering engine. Otherwise it schedules the crawl of the object, and once the object is crawled, it resends the content and crawl time of the web page to the rendering engine. The rendering engine receives the content and crawl time of a web page, requests all embedded objects, and renders the web page to an image once all embedded objects are received.

Iterative off-line rendering process

The technology described relates to reducing a backlog of pending URL crawls in view of a limited URL crawl capacity. This technology is useful for crawling URLs with low latency. Because of the limited crawl capacity, uncrawled URLs from crawl requests are entered into a backlog data structure of pending crawl requests. Various criteria are applied to the URL's that are requested to be crawled, so that less important URL crawls are rejected early from the backlog data structure. This early rejection tends to limit the backlog data structure to the more important pending URL crawls, and tends to keep the average latency low by quickly failing the less important requested URL crawls.

Method and apparatus for managing a backlog of pending URL crawls

A URL inspector may determine a uniform resource locator (URL) which includes an indexable fragment. A URL separator may separate, from the URL which includes the indexable fragment, a base URL occurring prior to the indexable fragment. An indexer may process content of the base URL to obtain processed content thereof. A rendering system may render the processed content together with the URL which includes the indexable fragment to obtain rendered content. A content converter may convert the rendered content into indexable content.

Indexing of URLs with fragments

A coverage evaluator determine search results that are a subset of potential search results, the search results provided in response to search requests, wherein a covered portion of the potential search results are associated with quick review information and a non-covered portion of the potential search results are not associated with quick review information. A coverage calculator may calculate a coverage of search results, based on a relation of covered, weighted impressions of the covered portion to weighted impressions, wherein the weighted impressions include non-covered, weighted impressions of the non-covered portion, if any.

Growing community of inventors

Menlo Park, CA, United States of America

Pawel Aleksander Fedorynski