Alta Vista Company

Hewlett-packard Development Company, L.p.

Guidewire Software, Inc.

Compaq Computer Corporation, Inc.

Clark Allan Heydon

Marc Alexander Najork

Monika Hildegard Henzinger

Butler W Lampson

Janet Lynn Wiener

Michael David Mitzenmacher

Kenneth William Branson

John Lome Seybold

Roy Levin

An insurance policy is stored () as a plurality of discrete temporally-sequential policy data revisions. A legally binding revision for a first given date is then determined () by identifying all policy data revisions effective on the first given date and choosing a most temporally recent policy data revision temporally prior to a second given date. When a new policy data revision is () temporally subsequent as compared to a first policy data revision and also comprises a legally effective date range preceding at least in part an effective date range of the first policy data revision, legally non-overlapping policy data revisions are created () for each legally overlapping effective date range as exists between the new policy data revision and all temporally preceding revisions. Each newly-created legally non-overlapping policy data revision comprises changes introduced by the new policy data revision and at least one temporally preceding policy data revision.

Insurance policy revisioning method and apparatus

A computing platform () of choice can be configured and arranged to access (and) a first memory () that stores attributes which specifically characterize a particular candidate insurance entity and a second memory () that stores mappings which relate insurance policy element availability for a plurality of insurance policy elements to various corresponding insurance entity characterizing attributes. The aforementioned attributes, by one approach, correspond to respective end user-configurable dimensions. This platform can then serve () as a matching component by automatically using the aforementioned mappings to determine available insurance policy elements for the particular candidate insurance entity as a function of these attributes and then automatically using () those available insurance policy elements to configure an insurance policy consistent with these mappings.

Method and apparatus to facilitate determining insurance policy element availability

A web crawler stores fixed length representations of document addresses in a buffer and a disk file, and optionally in a cache. When the web crawler downloads a document from a host computer, it identifies URL's (document addresses) in the downloaded document. Each identified URL is converted into a fixed size numerical representation. The numerical representation may optionally be systematically compared to the contents of a cache containing web sites which are likely to be found during the web crawl, for example previously visited web sites. The numerical representation is then systematically compared to numerical representations in the buffer, which stores numerical representations of recently-identified URL's. If the representation is not found in the buffer, it is stored in the buffer. When the buffer is full, it is ordered and then merged with numerical representations stored, in order, in the disk file. In addition, the document corresponding to each representation not found in the disk file during the merge is scheduled for downloading. The disk file may be a sparse file, indexed to correspond to the numerical representations of the URL's, so that only a relatively small fraction of the disk file must be searched and re-written in order to merge each numerical representation in the buffer.

System and method for efficient filtering of data set addresses in a web crawler

A system generates a list of near-uniform samples of data sets (e.g., web pages) from among a plurality of host computers. The system performs a random walk so as to generate a set of visited addresses. For each address in the set, a reachability measure is computed. Then, samples are selected from the set, such that the probability of selecting a given address is inversely proportional to the reachability measure for the address. The selected samples form the list of near-uniform samples.

System and method for near-uniform sampling of web page addresses

A method and system for scheduling downloads in a web crawler. A web crawler may use multiple threads to download documents from the world wide web. Both threads and queues are identified by numerical ID's. Each thread in the web crawler is assigned to dequeue from a queue until the assigned queue is empty. Each thread enqueues URL's as new URL's are discovered in the course of downloading web pages. In one embodiment, when a thread discovers a new URL, a numerical function is performed on the URL's host component to determine the queue in which to enqueue the new URL. In another embodiment, each queue in a web crawler may be dynamically assigned to a host computer so that URL's enqueued into the same queue all have the same host component. When a queue becomes empty, a new host may be dynamically assigned to it. In both embodiments, when all the threads are dequeuing in parallel from each of the respectively assigned queues, no more than one request to one host computer is made at the same time.

Web crawler system using parallel queues for queing data sets having common address and concurrently downloading data associated with data set in each queue

A web crawler downloads documents from among a plurality of host computers. The web crawler enqueues document addresses in a data structure called the Frontier. The Frontier generally includes a set of queues, with all document addresses sharing a respective common host component being stored in a respective common one of the queues. Multiple threads substantially concurrently process the document addresses in the queues. The web crawler includes a set of tools for storing an extensible set of data with each document address in the Frontier. These tools enable the applications to which the web crawler passes downloaded documents to store a record of information associated with each download, where each record of information includes an extensible set of name/value pairs specified by the applications. The applications also determine how many records of information to retain for each document, when to delete records of information, and so on. In another aspect of the present invention, the Frontier include a set of parallel &ldquo;priority queues,&rdquo; each associated with a distinct priority level. Queue elements for documents to be downloaded are assigned a priority level, and then stored in the corresponding priority queue. Queue elements are then distributed from the priority queues to a set of underlying queues in accordance with their relative priorities. The threads then process the queue elements in the underlying queues.

System and method for associating an extensible set of data with documents downloaded by a web crawler

A web crawler downloads data sets from among a plurality of host computers. The web crawler enqueues data set addresses in a set of queues, with all data set addresses sharing a respective common host address being stored in a respective common one of the queues. Each non-empty queue is assigned a next download time. Multiple threads substantially concurrently process the data set addresses in the queues. The number of queues is at least as great as the number of threads, and the threads are dynamically assigned to the queues. In particular, each thread selects a queue not being serviced by any of the other threads. The queue is selected in accordance with the next download times assigned to the queues. The data set corresponding to a data set address in the selected queue is downloaded and processed, and the data set address is dequeued from the selected queue. When the selected queue is not empty after the dequeuing step, it is assigned an updated download time. Then the thread deselects the selected queue, and the process of selecting a queue and processing a data set repeats. The next download time assigned to each queue is preferably a function of the length of time it took to download a previous document whose address was stored in the queue. For instance, the next download time may be set equal to the current time plus the a scaling constant multiplied by the download time of the previous document.

System and method for enforcing politeness while scheduling downloads in a web crawler

A web crawler stores fixed length representations of document addresses in first and second caches and a disk file. When the web crawler downloads a document from a host computer, it identifies URL's (document addresses) in the downloaded document. Each identified URL is converted into a fixed size numerical representation. The numerical representation is systematically compared to numerical representations in the caches and disk file. If the representation is not found in the caches and disk file, the document corresponding to the representation is scheduled for downloading, and the representation is stored in the second cache. If the representation is not found in the caches but is found in the disk file, the representation is added to the first cache. When the second cache is full, it is merged with the disk file and the second cache is reset to an initial state. When the first cache is full, one or more representations are evicted in accordance with an eviction policy. The representations include a prefix that is a function of a host component of the corresponding URL's, and the representations are stored in the disk file in sorted order. When the web crawler searches for a representation in the disk file, an index of the disk file is searched to identify a single block of the disk file, and only that single block of the disk file is searched for the representation.

System and method for efficient representation of data set addresses in a web crawler

A web crawler downloads documents from among a plurality of host computers. The web crawler enqueues document addresses in a data structure called the Frontier. The Frontier generally includes a set of queues, with all document addresses sharing a respective common host component being stored in a respective common one of the queues. Multiple threads substantially concurrently process the document addresses in the queues. The Frontier includes a set of parallel &ldquo;priority queues,&rdquo; each associated with a distinct priority level. Queue elements for documents to be downloaded are assigned a priority level, and then stored in the corresponding priority queue. Queue elements are then distributed from the priority queues to a set of underlying queues in accordance with their relative priorities. The threads then process the queue elements in the underlying queues. When performing a continuous crawl, the web crawler reinserts the queue element for a downloaded document into the Frontier in accordance with a download priority level associated with the downloaded document. For example, the download priority level may be determined as a function of an expiration date and time associated with document whose document address is denoted by the queue element.

Web crawler system using plurality of parallel priority level queues having distinct associated download priority levels for prioritizing document downloading and maintaining document freshness

Entries are cached in a function cache by statically assigning a primary key to each cache entry, and first grouping entries having identical primary keys, and dynamically assigning a secondary key to each cache entry, and then second grouping entries in each primary key group into sub-groups according to their secondary keys. The function cache is first accessed with a particular primary key to get the primary key group. Second, the primary key group is accessed with a particular secondary key to get a sub-group, and third, the sub-group is accessed with the same particular secondary key to get a matching cache entry.

Growing community of inventors

San Francisco, CA, United States of America

Clark Allan Heydon