Amazon Technologies, Inc.

Google Inc.

Peter Norvig

Anand Rajaraman

Ashish Gupta

Craig Nevill-Manning

Chade-Meng Tan

Manish Arya

Aynur Dyanik

Aynur Dayanik

Stephan Erickson

A rewrite component automatically generates rewrite rules that describe how uniform resource locators (URLs) can be rewritten to reduce or eliminate different URLs that redundantly refer to the same or substantially the same content. The rewrite rules can be applied to URLs received when crawling a network to increase the efficiency of the crawl and the corresponding document index generated from the crawl.

Automatic generation of rewrite rules for URLs

A wrapper builder application provides a variety of features for facilitating the creation of wrappers that are used to extract information from Web sites. In one aspect, the wrapper builder application provides a tool with which the process of creating a wrapper, which typically resembles coding, can be accomplished by a graphical design process involving drag and drop operations, clicking on objects, and filling in forms. A web viewer component provides a web browser frame, a source code frame, and a tree view frame, enabling the user to identify semistructured information of interest on Web sites. A wrapper editor component provides a graphical design environment in which a wrapper can be graphically constructed from operations and links. A wrapper model component provides a functioning internal representation of the graphically designed wrapper using Java objects and methods. A property editor component provides for the setting of properties that define the particular functionality of individual wrapper operations. A wrapper execution component provides features that enable the wrapper to be executed and debugged using a number of debugging tools. A wrapper serialization component provides a mechanism for storing and retrieving a wrapper for subsequent use and/or modification.

Software application and associated methods for generating a software layer for structuring semistructured information

According to the invention, a system and method for extracting information from a semistructured information source. The system includes a listing stack for holding extracted information. A means for matching at least one extractor to the semistructured information to return a list of potential matches is also included. The system can also include a means for iterating through the list of potential matches and a means for retrieving information from a particular match in the list of potential matches. A means for adding a particular match into the listing stack can also be part of the system.

Method and apparatus for creating extractors, field information objects and inheritance hierarchies in a framework for retrieving semistructured information

According to the invention, a method is provided for forming an information closure of a plurality of rows in a listing stack built by a wrapper program for accessing semistructured information. This method includes removing a first row from the listing stack and computing a cross product of the fields in the first row. A step of adding this cross product to a list of accepted rows can also be part of the method. For each remaining row in the listing stack, the method includes a step of computing a selective cross product according to a plurality of steps. In one step, a result is initialized to empty. Then, for each row in the list of accepted rows, a step of determining for a first new row from the accepted row, extended with the non-empty fields of the remaining row is performed. The method can also include a step of determining a second new row from the remaining row, extended with the non-empty fields in the accepted row. Thereupon, a step of adding the two new rows to the result can be performed. Repeating the determining steps and the adding step for all rows in the list of accepted rows, and removing from the result any identical rows can provide an information closure.

Method for creating an information closure model

According to the invention, methods for gathering data around forms having one or more fields, enabling a wrapper program to extract semistructured information by determining combinations of values for fields associated with particular forms; submitting the particular forms repeatedly for all combinations of interest and providing the results returned for further processing. In select embodiments, the combinations of values for fields is a Cartesian product of the possible values for the fields. Values to be submitted in the form fields may be specified by using a programming language such as Site Description Language (SDL) or Java.

Growing community of inventors

Palo Alto, CA, United States of America

Peter Norvig