Method for analyzing data including a large number of individual messages, computer program product, and computer system

Abstract

In order to be able to confidently interpret log files of various system domains, experts need to undergo extensive training. Knowledge about one system does not necessarily transfer to other systems. Domain experts manually inspect log files, trying to find anomalies and concerning events. In some cases, domain experts may be assisted by log monitoring tools that match hand-crafted regular expressions against log entries or are use regular expressions to cluster log entries into groups. The proposed method advantageously identifies sequence patterns from semi-structured text log messages and classifying the identified patterns (e.g., blocks or a portion of a block) into categories (e.g., error, warning, and info). The identified patterns represent higher-level events that happen in the industrial plant (e.g., a plant-wide update process) and important sections may be automatically highlighted in a log file to human reviewers/domain experts.