Splunk Inc.

Splunkinc.

David Marquardt

Stephen Phillip Sorkin

Steve Yu Zhang

Mitchell Blank, Jr

Karthikeyan Sabhanatarajan

Jindrich Dinga

Jason Szeto

Jose Solis

Jesse Brandau Miller

Alexander Douglas James

Hailun Yan

Christopher Madden Pride

Vishal Patel

Ledion Bitincka

Thomas Allan Haggie

Clint Sharp

Nicholas Robert Romito

Xiaowei Wang

Elizabeth Lin

Kishore Reddy Ramasayam

David Everett Simmen

Igor Stojanovski

Murugan Kandaswamy

Marc Andre Chene

Sophia Rui Zhu

Amrittpal Singh Bath

Pratiksha Shah

Esguerra Ma Kharisma

Jesse Miller

Systems and methods are disclosed involving user interface (UI) search tools for locating data, including tools for summarizing indexed raw machine data that organize and present results to enable expansion and exploration of initial summarizations. The initial summarizations may be explored and refined to help users determine how to identify and best focus a search on data subsets of greater interest.

Data identification using inverted indexes

Embodiments are directed are towards a method for generating a query response, which comprises creating two or more partitions of event records from raw data stored in a data store, wherein each event record in the two or more partitions of event records includes a portion of the raw data and is associated with a time stamp derived from the raw data. The method also comprises generating a summarization table for each partition of the two or more partitions that: (a) identifies a field value comprising a value that corresponds to an associated field extracted from a respective event record; and (b) for the field value, includes a posting value to the respective event record within a respective partition. The method further comprises generating partial results for a received query using summarization tables in the partitions and generating a response to the query by combining the partial results.

Generating a query response by combining partial results from separate partitions of event records

Embodiments of the present disclosure provide techniques for using an inverted index in a pipelined search query. A field searchable data store is provided that comprises a plurality of event records, each event record comprising a time-stamped portion of raw machine data. Responsive to the reciept of an incoming search query, the search engine accesses an inverted index, wherein each entry in the inverted index comprises at least one field name, a corresponding at least one field value and a reference value associated with each field name and value pair that identifies a location in the data store where an associated event record is stored. Once the inverted index is accessed, it can be used to identify and search a subset of the plurality of event records, wherein the subset comprises one or more event records with corresponding reference values in the inverted index.

Pipelined search query, leveraging reference values of an inverted index to access a set of event data and performing further queries on associated raw data

Embodiments of the present disclosure provide techniques for performing searches of event records by leveraging reference values in an inverted index. A method of searching comprises accessing a query associated with a first set of event records in a field searchable data store, each event record comprising a time-stamped portion of raw machine data. The method further comprises evaluating the query and generating results for the query by accessing an inverted index, wherein each entry in the inverted index comprises at least one field, a corresponding at least one field value and a reference value that identifies a location in the field searchable data store where an associated event record is stored. The method further comprises performing a search to filter out a second set of event records and retrieving the second set of event records from the field searchable data store using reference values in the inverted index.

Retrieving event records from a field searchable data store using references values in inverted indexes

Embodiments are directed towards a method for searching data. The method comprises providing an inverted index that comprises at least one record, wherein the at least one record comprises at least one field name and a corresponding at least one field value. The at least one field name and corresponding value are extracted from time-stamped searchable events that are stored in a field searchable datastore and comprise portions of raw data. The at least one record further comprises a posting value that identifies a location in the field searchable datastore where an event associated with the at least one record is stored. The method further comprises evaluating an incoming search query that references a field name. Responsive to the evaluating, the method comprises determining results for the incoming search query by executing the incoming search query across the field searchable datastore or the inverted index.

Query handling using a field searchable datastore or an inverted index

Data categorization using inverted indexes

Provided are systems and methods for concurrent summarization of indexed data. In some embodiments, two or more summary processes can be executed concurrently (e.g., in parallel) by an indexer to generate summaries for respective subsets of indexed data (e.g., partitions or buckets of indexed data) managed by the indexer.

Systems and methods for scheduling concurrent summarization of indexed data

A method includes receiving an initial pipeline including a sequence of commands for execution on a computing system, and obtaining, for each command in the sequence of commands, semantic information. The sequence of commands includes a command with incomplete semantic information. The method further includes generating an abstract semantic tree (AST) with the semantic information and a placeholder for the incomplete semantic information, and manipulating the AST to generate a revised AST. The revised AST corresponds to a revised pipeline that reduces an execution time on the computing system. The method further includes executing the revised pipeline.

Query execution optimization for partially typed semantics

Embodiments of the present disclosure provide techniques for using an inverted index in a pipelined search query. A field searchable data store is provided that comprises a plurality of event records, each event record comprising a time-stamped portion of raw machine data. Responsive to the receipt of an incoming search query, the search engine accesses an inverted index, wherein each entry in the inverted index comprises at least one field name, a corresponding at least one field value and a reference value associated with each field name and value pair that identifies a location in the data store where an associated event record is stored. Once the inverted index is accessed, it can be used to identify and search a subset of the plurality of event records, wherein the subset comprises one or more event records with corresponding reference values in the inverted index.

Pipelined search query, leveraging reference values of an inverted index to determine a set of event data and performing further queries on the event data

Embodiments of the present disclosure provide techniques for emitting structured and dynamic fields from an accelerated data model. The method comprises evaluating a query to search a data model, wherein the data model is defined by a set of events and at least one structured field from fields associated with the set of events. Each event comprises a time-stamped portion of raw machine data and is stored in a field searchable data store. A summarization table is associated with the data model and comprises a plurality of entries comprising reference values, wherein a respective summarization table entry comprises: the at least one structured field; a respective field value; and a reference value. The method further comprises accessing the set of events from the field searchable data store using the reference values in the summarization table and annotating the set of events with the at least one structured field and with at least one dynamic field from the fields associated with the set of events, wherein the at least one dynamic field is not defined in the data model.

Optimizing search of an accelerated data model by enabling emitting of structured and unstructured fields from the data model

User interface search tool for identifying and summarizing data

Leveraging references values in inverted indexes to retrieve associated event records comprising raw machine data

The disclosed embodiments include a method performed by a data intake and query system. The method includes populating each metric including a measure value, cataloging metadata in an in-memory metrics catalog, where the metadata is related to the metrics. The method further includes receiving a search query including search criteria, evaluating the search query by applying the search criteria to the metadata of the metrics catalog to obtain results that satisfy the search criteria, and causing display, on a display device, of the results or data indicative of the results.

In-memory catalog for searching metrics data

The disclosed embodiments include a method performed by a data intake and query system to store and query metrics data. The method includes ingesting metrics, where each metric includes key values and numerical value indicative of a measured characteristic of a computing resource. The method further includes populating a first portion of a metric-series index (msidx) file with the key values and a second portion of the msidx file with numerical values indicative of a measured characteristic, where the first portion is distinct from the second portion. The method further includes receiving a query including criteria, evaluating the query by applying the criteria to the first portion of the msidx file to obtain query results indicative of metrics that satisfy the criteria, and displaying, on a display device, the query results or data indicative of the query results.

Storing and querying metrics data using a metric-series index

The disclosed embodiments include a method performed by a data intake and query system. The method includes ingesting each metric including at least one key value and a measured value taken of a computing resource, and storing each metric in an index of a metrics store, where the index defines at least one dimension populated with the at least one key value and a measure populated with the measured value. The method further includes cataloging metadata in a metrics catalog, where the metadata is related to the metrics stored in the metrics store, performing an analysis of metrics data included in the metrics store and/or the metrics catalog to obtain results, and causing display of the results or an indication of the results on a display device.

Growing community of inventors

San Francisco, CA, United States of America

David Marquardt