International Business Machines Corporation

Lu An

Anbang Xu

Gary I Givental

Aankur Bhatia

Harshit Kumar

Xiaotong Liu

Sahil Bansal

Ruchi Mahindru

Rama Kalyani T Akkiraju

Gargi Banerjee Dasgupta

Neil Boyette

An-Jie Andy Tu

A computer-implemented method, a computer program product, and a computer system for log anomaly detection. A computer receives a windowed log of incoming raw log messages. A computer compares statistical distribution metrics of entities in the windowed log with a statistical distribution extracted from a real-time statistical model for the entities. In response to the statistical distribution metrics being statistically different from the statistical distribution extracted from the real-time statistical model for the entities, a computer tags the windowed log as an entity anomaly. A computer computes a distance between an average word embedding vector in the windowed log and a statistical distribution extracted form a real-time statistical model for word embeddings. In response to the distance being greater than a predetermined threshold, a computer tags the windowed log as a word embedding anomaly. A computer sends to a user an alert with an anomaly severity level.

Log anomaly detection in continuous artificial intelligence for it operations

Identifying an log anomaly resolution by generating a knowledge base linking each of a plurality of incidents with historical anomalous log lines, calculating a resolution specificity score for each knowledge base record, identifying a run-time anomalous log line using the knowledge base, predicting a category for the run-time anomalous log line, identifying resolutions according to the category, ranking the resolutions according to the resolution specificity scores, and recommending a resolution according to the ranking.

Identifying log anomaly resolution from anomalous system logs

One or more computer processors classify each log line in a plurality of unlabeled log lines as an erroneous log line or a non-erroneous log line. The one or more computer processors templatize each classified erroneous log line and non-erroneous log line in the plurality of unlabeled log lines. The one or more computer processors cluster erroneous log templates into erroneous log template clusters and the non-erroneous log templates into non-erroneous log template clusters. The one or more computer processors eliminate the erroneous log template clusters and the non-erroneous log template clusters that exceed a frequency threshold. The one or more computer processors train a log anomaly model utilizing=remaining erroneous log template clusters and remaining non-erroneous log template clusters. The one or more computer processors identify a subsequent log line as anomalous or non-anomalous utilizing the trained log anomaly model.

Unlabeled log anomaly continuous learning

Mechanisms are provided to implement an ensemble of unsupervised machine learning (ML) models. The ensemble of unsupervised ML models processes a portion of input data to generate an ensemble output and the ensemble output is output to an authorized user computing device to obtain user feedback from the authorized user via the user computing device. The user feedback indicates a correctness of the ensemble output. The mechanisms modify at least one feature of the ensemble of unsupervised ML models based on the obtained user feedback to thereby generate a modified ensemble of unsupervised ML models. Subsequent portions of input data are then processed using the modified ensemble of unsupervised ML models.

Modification of machine learning model ensembles based on user feedback

Mechanisms are provided to implement a hybrid machine learning (ML) anomaly detector comprising an ensemble of unsupervised ML models and a semi-supervised ML model. The ensemble of unsupervised ML models are executed on log data to generate, for each entry in the log data, a predicted anomaly score and corresponding anomaly classification label of the entry. A partially labeled dataset is generated based on a selected subset of entries and other unlabeled log data in the log data. A similarity analysis of the unlabeled log data with entries in the selected subset of entries is performed and anomaly classification labels of the selected subset of entries are propagated to the other unlabeled log data based on the similarity analysis.

Growing community of inventors

Raleigh, NC, United States of America

Lu An