Google Inc.

Cheng Xu

Xin Li

Gang Feng

Qiying Lin

Methods, systems and computer-readable storage medium for determining a crawling schedule. In an aspect, a method includes obtaining crawl history data for a Web site having Web pages, determining a status of the Web pages, determining a total quantity of Web pages that have a status of deleted, calculating a probability that another Web page of the Web site will be removed based on the total quantity, and storing data associating the calculated probability with the Web site. The method can further include determining, for a plurality of sets of the previous time periods, a respective crawl penalty as a combination of a penalty for crawling the Web site and a penalty for showing a deleted Web page based on the calculated probability, and determining a re-crawl schedule based on the crawl penalties.

Method and techniques for determining crawling schedule

Systems, methods, and computer readable storage mediums are provided to automatically identifying a classified website. A website is determined to be a candidate site based on a set of heuristics. From among pages constituting the candidate site one or more pages are determined to be listing page candidates and one or more pages are determined to be detail page candidates. Then a listing page score is determined using a listing page classifier. Similarly, a detail page score is determined using a detail page classifier. The listing page and detail page scores each indicate the likelihood that the pages are part of a classified website. A candidate site score is determined based in part on a combination of the listing page score and the detail page scores. Then when the candidate site score is above a threshold the candidate site is determined to be a classified website.

System and method for automatically identifying classified websites

Systems, methods, and computer readable storage mediums are provided for automatically extracting data from a classified website. A website is determined to be a classified website based on a set of heuristics. Then page models for other classified websites are accessed. The page models may include listing page models, detail page models, and/or city page models. A listing page in the classified website is determined based on similarity of the listing page to the page models for the other classified websites. Then a listing page model for the listing page in the classified website is created. After the model has been created data from the classified website is extracted based at least in part on the listing page model. Similar processes are performed for determining a details page, creating a details page model, and extracting data from the classified website using a details page model.

Growing community of inventors

Sunnyvale, CA, United States of America

Cheng Xu