Metacluster Lt, Uab

Oxylabs, Uab

Rimantas Stankevicius

Martynas Juravicius

Giedrius Stalioraitis

Eivydas Vilcinskas

Arnas Petruskevicius

Arnas Petruškevicius

Embodiments disclose a system that allows for improved generation of web requests for scraping that, because of the nature of the requests and time and manner they are sent out, appear more organic, as in human generated, than conventional automated scraping systems. The system then manages how a client request to scrape a target website is made to the site, masking the request in a manner that makes it appear to the Web server as if the request is not generated by an automated system. In this way, by appearing more organic, Web servers may be less likely to block requests from the disclosed system or may take longer to block requests from the disclosed system. By avoiding Web servers blocking requests and extending the lifetime of IP proxies before they are blocked, embodiments can use a limited IP proxy address space more efficiently.

Token-based authentication for a proxy web scraping service

Direct leg access for proxy web scraping

Direct data center request for proxy web scraping

Database server management for proxy scraping jobs

Web scraper history management across multiple data centers

Throttling client requests for web scraping

Growing community of inventors

Vilnius, Lithuania

Rimantas Stankevicius