Johns Hopkins University

Jacob William Green

John Lane Schultz

Yair Amir

Michael Truman Goodrich

A system for maximal gathering of fresh information added to a network such as the as the Internet and for processing the gathered fresh information. A link server () sends a batch of links to check () to a crawler (B). Crawler (B) them executes its crawling assignment by filtering the encountered content and extracting only that which is new or changed (). Crawler (B) then returns this content () to at least one data center and any interested web mining application (). By using the crawlers (A-E) to filter the data and only return or notify regarding, the fresh content, less bandwidth is needed to get the information to the web mining application ().

High refresh-rate retrieval of freshly published content using distributed crawling

A dynamic-content web crawler is disclosed. These New Crawlers (NCs) are located at points between the server and user, and monitor content from said points, for example by proxying the web traffic or sniffing the traffic as it goes by. Web page content is recursively parsed into subcomponents. Sub-components are fingerpinted with a cyclic redundancy check code or other loss-full compression in order to be able to detect recurrence of the sub-component in subsequent pages. Those sub-components which persist in the web traffic, as measured by the frequency NCs () are defined as having substantive content of interest to data-mining applications. Where a substantive content sub-component is added to or removed from a web page, then this change is significant and is sent to a duplication filter () so that if multiple NCs () detect a change in a web page only one announcement of the changed URL will be broadcast to data-mining applications (). The NC () identifies substantive content sub-components which repeatably are part of a page pointed to by a URL. Provision is also made for limiting monitoring to pages having a flag authorizing discovery of the page by a monitor.

Growing community of inventors

Bloomfield Hills, MI, United States of America

Jacob William Green