Determining whether to process identified uniform resource locators

Abstract

In one general embodiment, a computer program product for determining whether to process a uniform resource locator (URL) comprises a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, and where the program instructions are executable by a processor to cause the processor to perform a method comprising selecting, by the processor, a portion of a hash table, identifying, by the processor, a plurality of uniform resource locators (URLs) stored within the selected portion of the hash table, and for each of the plurality of identified URLs, individually determining, by the processor, whether to process the identified URL, based on data associated with the identified URL.