Splunk Inc.

Cisco Technology, Inc.

Zhaohui Wang

Ram Sriharsha

Abhinav Mishra

Kristal Curtis

Xiao Lin

Chandrima Sarkar

Ryan Gannon

Abraham Starosta

Giovanni Mola

A computerized method for detection of format drift and format anomalies is described. A format representation for each data point of a first data sample is extracted. Transformations of each format representation is conducted, resulting in a first plurality of count values (reference) and a second plurality of count values. Each count value identifies a number of occurrences of a transformed format representation within that data sample. Thereafter, a first probability distribution for the first plurality of count values and a second probability distribution for the second plurality of count values are computed. Analytics using the first and probability distributions are conducted to produce a first metric. A format drift is determined based on an evaluation of the first metric to a second metric operating as a threshold metric. Format anomalies are detected based on analytics of hashed format representation and determination of infrequent usage of a particular format representation.

System and method for format drift and format anomaly detection

Systems and methods are described for extracting data fields from logs ingested in a data processing pipeline or otherwise stored. For example, a log can be applied as an input to an artificial intelligence model trained to infer a log sourcetype of logs, and the artificial intelligence model can output an inferred log sourcetype of the log. The inferred log sourcetype can be used to select another artificial intelligence model trained to extract data fields from logs having the inferred log sourcetype, and the log can then be applied as an input to the other artificial intelligence model. The other artificial intelligence model may then output one or more data fields extracted from the log.

Data field extraction by a data intake and query system

A computerized method is disclosed that includes operations of training a machine learning model using a labeled training set of data, wherein the machine learning model is configured to classify domain name server (DNS) records, obtaining DNS record data including at least a first DNS Txt record, applying the trained machine learning model to the first DNS Txt record to classify the first DNS Txt record and responsive to the classification of the first DNS Txt record, generating a flag for a system administrator. The trained machine learning model may classify the first DNS Txt record using logistic regression. In some instances, applying the trained machine learning model to the first DNS Txt record includes performing a tokenizing operation on the first DNS Txt record to generate a tokenized first DNS Txt record.

Systems and methods for DNS text classification

A computerized method for detection of categorical drift within an incoming data stream. Herein, an error threshold is computed based on a first set of training data samples selected to detect categorical drift occurring for a data stream. Thereafter, probability distributions associated with content of a first and second data samples of the data stream are computed. Analytics are conducted to compute a difference between content of the first probability distribution that is based on a first data point of the first data sample and content of the second probability distribution that is based on a first data point of the second data sample. After computing the difference, that categorical drift is determined whether categorical drift detection has been conducted.

System and method for categorical drift detection

A computerized method is disclosed including operations of receiving a data stream, performing a changepoint detection resulting in a detection of changepoints in the data stream including: maintaining a listing of starting indices for each run within the data stream in a buffer of size L wherein each index of the listing has a run length probability representing a likelihood of being a changepoint, receiving a new data point within the data stream and adding a new index to the buffer resulting in the buffer having size L+1, calculating a posterior run length probability that the new data point is a changepoint, and removing an index from the listing that has a lowest run length probability thereby returning the buffer to size L, and responsive to determining the index removed from the listing does not correspond to the new data point, identifying a changepoint associated with the new data point.

System and method for changepoint detection in streaming data

The disclosure provides implementations for determining whether domain name server (DNS) beaconing is present within a communication session. Some implementations provide a method that includes multiple analyses directed to analyzing each of a time-to-live (TTL) run length distribution for a plurality of DNS records within the communication session and analyzing whether the communication is comprised of at least a threshold number of transmissions. As used in the analyses, the communication session may be comprised of transmissions between a first source device and a first DNS. When DNS beaconing is detected within the communication session, some implementations of the disclosure provide for generating an alert to an administrator or other user.

Detection of DNS beaconing through time-to-live and transmission analyses

Systems and methods are described for training an artificial intelligence model to infer a log sourcetype of a log. For example, logs may have different log sourcetypes, and logs having the same log sourcetypes may have different messagetypes. The artificial intelligence model may be a machine learning model, and can be trained using training data that includes logs with known log sourcetypes. Each log can be tokenized, filtered, converted into a vector, and applied to a machine learning model as an input to perform the training. The machine learning model may output an inferred log sourcetype, which can be compared with the known log sourcetype to update model parameters to improve the machine learning model accuracy. The trained machine learning model may be trained to infer a log sourcetype of a log regardless of the messagetype of the log.

Log sourcetype inference model training for a data intake and query system

Systems and methods are described for training an artificial intelligence model to extract one or more data fields from a log. For example, the artificial intelligence model may be a neural network. The neural network may be trained using training data obtained by iterating through a plurality of logs using active learning, and selecting a subset of the logs in the plurality to be labeled by a user. For example, the selected subset of logs may be logs that are not similar to other logs already labeled by a user. The user may be prompted to label the selected subset of logs to identify one or more data fields to extract. Once the selected subset of logs are labeled, these labeled logs can be used as the training data to train the neural network.

Data field extraction model training for a data intake and query system

A computerized method is disclosed that includes accessing domain name server (DNS) record data including a plurality of DNS records spanning a first time period, performing a time-to-live (TTL) analysis to determine a TTL run length distribution for the DNS record data, wherein the TTL analysis includes: generating a vector of the TTL values of each DNS record ordered sequentially in time, parsing the vector of the TTL values into segments, where a segment consists of one or more TTL values where a current TTL value is less than an immediately preceding TTL value, and determining the TTL run length distribution, determining whether DNS beaconing is present based on a result of the TTL analysis and in response to determining that DNS beaconing is present, generating an alert for a system administrator.

Growing community of inventors

San Francisco, CA, United States of America

Zhaohui Wang