Metacluster Lt, Uab

Oxylabs, Uab

Metacluster It, Uab

Eivydas Vilcinskas

Martynas Juravicius

Giedrius Stalioraitis

Rimantas Stankevicius

Arnas Petruskevicius

Arnas Petruškevicius

Embodiments disclose a system that allows for improved generation of web requests for scraping that, because of the nature of the requests and time and manner they are sent out, appear more organic, as in human generated, than conventional automated scraping systems. The system then manages how a client request to scrape a target website is made to the site, masking the request in a manner that makes it appear to the Web server as if the request is not generated by an automated system. In this way, by appearing more organic, Web servers may be less likely to block requests from the disclosed system or may take longer to block requests from the disclosed system. By avoiding Web servers blocking requests and extending the lifetime of IP proxies before they are blocked, embodiments can use a limited IP proxy address space more efficiently.

Token-based authentication for a proxy web scraping service

Direct leg access for proxy web scraping

Direct data center request for proxy web scraping

Systems and methods of task implementation are extended as provided herein and target the web crawling process through a step of submitting a request by a customer to a web crawler. The systems and methods allow a more complex request for a web crawler to be defined in order to receive more specific data. In one aspect, a method for data extraction and gathering from a Network by a Service provider infrastructure include the following steps: checking the parameters of a request received from a User's Device, adjusting the request parameters according to pre-established Scraping logic, selecting a Proxy according to the criteria of the pre-established Scraping logic, sending the adjusted request to the Target through the selected Proxy, checking metadata received from the Target, and forwarding the data to the User's device.

Dynamic optimization of request parameters for proxy server

Database server management for proxy scraping jobs

Web scraper history management across multiple data centers

Throttling client requests for web scraping

The task, logic of HTTP/HTTPS session statistics interception and collection is moved to the client side instead of the proxy layer. Encrypted HTTPS tunnel is terminated at the client end, making the actual content or data in transit invisible to both proxies and the smart proxy rotator (SPR). Client's scraping software has a plug-in installed that expands its functionality. HTTP/HTTPS session quality metrics are intercepted and collected at the client side, then sent to the SPR. Proxy usage mark 'can be used' is obtained from the SPR for the currently analyzed proxy, based on the results of metrics analysis.

Smart proxy rotator

Growing community of inventors

Siauliai, Lithuania

Eivydas Vilcinskas