Splunk Inc.

Lin Ma

Ram Sriharsha

Mohan Rajagopalan

John C Reed

Amir Malekpour

Chinmay Madhav Kulkarni

Adam Jamison Oliner

Jacob Barton Leverich

Hongyang Zhang

Alex Cruise

Vishal Patel

Itay Alfred Neeman

Glenn Block

Mitch Blank

Frank Ye

An interface and improved data intake and query system is described herein that allows users to define metrics and that aggregates metric values regardless of the level at which a metric is defined and/or the level at which metric values are available. The improved data intake and query system can initialize a sketch in response to a user providing one or more metric definitions. The initialized sketch includes one or more instances, where each instance produces an output and collects metric value(s), appends the metric value(s) to the output, and forwards the appended data to a process function downstream in a data processing pipeline. The process function separates the output and the metric value(s), sending the output further downstream in the data processing pipeline and sending the metric value(s) to a parallel process function that sits outside the data processing pipeline. The parallel process function can persist the metric value(s).

Metric processing for streaming machine learning applications

A method for deployment of machine-learning based operators within a query is described. For this embodiment, a sequence of operators associated with a query is identified, which includes at least a first operator and at least a second operator. The second operator is configured to perform operations, in accordance with a machine learning (ML) component, on data received as input from execution of the first operator. Schemas associated with the machine learning component is retrieved along with schemas associated with other operators within the sequence. Compatibility between at least an output schema associated with the first operator and an input schema associated with the second operator associated with the ML component is determined. Thereafter, a portion of the sequence of operators including at least the second operator and another operator of the sequence of operators successive to the second operator may be stored within a data store for subsequent use.

Systems and methods for auto-deployment of a machine learning component within a pipelined search query

Disclosed herein is a method that supports queries deploying operators based on multiple programming languages at least through determining schema compatibility between neighboring operators within a query. Upon receipt of a query, a sequence of operators of the query is identified, where the sequence of operators includes at least two neighboring operators including a first operator and a second operator representing a machine learning model. By determining schema compatibility between at least the first and second operators, the method either alerts a user to schema incompatibility before attempting to execute the query or determine that the schemas are compatible such that the query may be executed without the occurrence of errors due to schema incompatibility between neighboring operators. Advantageously, the method enables the integration of a machine learning model into the query while still ensuring schema compatibility.

Determination of schema compatibility between neighboring operators within a search query statement

A data intake and query system receives a message including raw machine via an internet protocol (IP) such as the hypertext transfer protocol (HTTP). The message includes a distinct payload portion and a distinct custom field portion. The payload portion includes raw machine data, while the custom field portion includes values for fields. An event that includes the raw machine data and the values is generated from the payload portion and the values are extracted from the custom field portion. The event is then stored such that the values are associated with the event.

HTTP events with custom fields

A computer-implemented method is disclosed that includes operations of parsing a query comprised of a sequence of operators to detect each operator of the sequence of operators, where the sequence of operators includes a machine learning (ML) operator representing a trained ML model. Additionally, a schema of the ML operator is determined through metadata. A filter or a projection is generated based on the schema of the ML operator, where the filter or projection is configured to reduce an amount of data retrieved upon application of the filter of the projection to an operator of the sequence of operators comprising the query. The schema of the ML operator indicates a schema of input data to be provided to the ML operator and a schema of output data to be provided by the ML operator following processing.

Systems and methods for decoupling search processing language and machine learning analytics from storage of accessed data

A computer-implemented method for integration of machine learning components within a pipelined search query to generate a visualization is described. Herein, an interface is provided for receipt of pipelined code into a web-based programming application. The pipelined code features a series of operators configured to perform one or more tasks based on collective operations by the series of operators, wherein a first operator of the series of operators is to receive input data from a selected data source and each remaining operator of the series of operators to receive input based on an output from a preceding operator of the remaining operators. The task(s) performed by the pipelined code generate results including visualizations. The visualization is rendered in a manner that allows the pipelined code to be scrolled to display the pipelined code or the visualization.

Systems and methods for integration of machine learning components within a pipelined search query to generate a graphic visualization

Embodiments of the present invention are directed to facilitating distributed data processing for machine learning. In accordance with aspects of the present disclosure, a set of commands in a query to process at an external computing service is identified. For each command in the set of commands, at least one compute unit including at least one operation to perform at the external computing service is identified. Each of the at least one compute unit associated with each command is analyzed to identify an optimized manner in which to execute the set of commands at the external computing service. An indication of the optimized manner in which to execute the set of commands and a corresponding set of data is provided to the external computing service to utilize for executing the set of commands at the external computing service.

Efficient command execution using aggregated compute units

A computer-implemented method is disclosed including operations of receiving a request to store a representation of a machine learning model in a non-transitory computer-readable medium, validating the representation of the machine learning model, storing the representation of the machine learning model, receiving a query from a web-based programming application, the query including a sequence of operators, parsing the query to detect and identify each operator within the sequence of operators, converting the query to directed acyclic graph (DAG) and providing the DAG to a distributed processing engine configured to execute the DAG. The computer-implemented method includes further operations of, prior to converting the query to the DAG, altering the query to improve efficiency of execution of the DAG. Altering the query may include at least one of consolidating at least two operators, applying a filter operation to an operator, or applying a projection to the operator.

Systems and methods for a unified analytics platform

According to one embodiment, a method that supports queries deploying operators based on multiple programming languages is described. A sequence of operators associated with a query is identified, where the sequence of operators includes at least two neighboring operators including a first operator based on a first programming language and a second operator based on a second programming language that is different from the first programming language. Thereafter, a schema associated with the first operator and a schema associated with the second operator is determined along with the compatibility between the schema of the first operator and the schema of the second operator. A query error message is generated in response to incompatibility between the first operator schema and the second operator schema. Compatibility is determined when an output generated by execution of the first operator provides machine data needed as input for execution of the second operator.

Systems and methods for integration of multiple programming languages within a pipelined search query

Efficiently executing commands at external computing services

A computer-implemented method is disclosed that includes operations of receiving a query to be executed, the query including an indication of a data source at which input data is be to obtained, wherein the query is to be executed on the input data, determining a schema of the input data, determining fields of the input data that are required for execution of the query by analyzing a sequence of operators forming the query, determining one or more alterations to the query to improve efficiency of the execution of the query based on the fields of input data required for the execution, and generating an altered query be altering the query in accordance with the one or more alterations. The method may further include converting the query to a directed acyclic graph (DAG) and providing the DAG to a distributed processing engine configured to execute the DAG.

Growing community of inventors

Vancouver, Canada

Lin Ma