Automated search evaluation

Abstract

Methods and apparatuses for dynamically generating and applying automated search evaluation sets to improve search results and to automatically detect and correct search system issues are described. A search evaluation set may comprise a set of search evaluation vectors that each map a search query and corresponding properties of the search query to a canonical search result. To detect search system issues over time, baseline search result rankings may be periodically generated or automatically generated after software updates to the search system have been made. Two consecutive baseline search result rankings using the same search evaluation set may then be compared to detect result deviations in search result rankings. If the search system detects that at least a threshold number of result deviations have occurred, then the search system may perform subsequent actions to automatically detect and correct search system issues.