System and method for dynamically limiting robot access to server data

Abstract

A method for automatically limiting access of a client computer to data objects accessed through a server computer dynamically prevents robots or webcrawlers from obtaining too much of the server database and from dramatically reducing server performance. The method includes the steps of receiving a request for a data object, recording a log entry for the request, calculating client request values, and refusing the request if a client request value exceeds one of a set of corresponding predefined maximum request values. Each log entry contains a client identifier, timestamp, and at least one data object identifier for the request. The client request values preferably include a request frequency, which is compared with a predefined maximum request frequency, and a cumulative data request, which is compared with a data access threshold. If the client is refused access, the client identifier is added to a deny list, and future requests from the client are automatically denied. The calculated cumulative data request may be for a single client, or it may be for all clients, in order to detect a robot that is divided among multiple client identifiers. The cumulative data request check may consider the total percentage of server resources being given away, or a pattern in the requests. Also provided is a data protection system containing a log file, a request analyzer, and a dynamically-generated deny list. Requests to the server are intercepted and sent to the data protection system first.