Splunk Inc.

Junqing Hao

Ting Wang

Gang Tao

Elias Haddad

John Robert Coates

Takuya Matsumoto

Guodong Wang

Yongxin Su

Lai Qiang Ding

Dritan Bitincka

Ken Chen

Ziliang Chen

Determining a set of extraction rules include clustering event segments into at least a first group of event segments, and determining, using first field data in the first group of event segments, a first set of extraction rules for extracting the first field data from each event segment of the first group of event segments. A determination is made that the first set of extraction rules fails to successfully extract all of the first field data. Responsive to the determination, the event segments are re-clustered into at least a second group of event segments and a third group of event segments until a successful set of extraction rules are identified. The successful set of extraction rules are stored in computer memory.

Extraction rule generation using clustering

The operation of an automatic data input and query system is controlled by well-defined control data. The system exposes user interfaces enabling an administrator to interact with control data to modify the ongoing operation of the system. Certain control data determines the collection and treatment of data from various technology sources. A robust control interface is provided enabling the efficient and reliable adding on of new technology data sources. Once established, control data for a new technology data source may be packaged in a form for archiving or distribution. The system may support the export and import of such packages. Such packages may be created independently of the system.

Technology add-on interface

Technology add-on control console

The operation of an automatic data input and query system is controlled by well-defined control data. Certain control data may relate to data schemas and direct operations performed by the system to extract fields from machine data. Automatic methods may determine proper field extraction control information by analyzing a sample of data from a source, breaking the sample data into event segments, classifying the segments into groups based on a measure of similarity, determining an operable extraction rule for each group, and storing the resulting extraction model. Data patterns known by the system can be leveraged to perform the event breaking and field identification for the classifying. Embodiments may provide a user interface to view, interact with, and approve the computer-generated extraction model.

Field extraction rules from clustered data samples

Techniques and mechanisms are disclosed that enable a data collection system to adaptively control collection of data from one or more external data sources. At a high level, adaptively controlling collection of data from external data sources may include collecting performance information related to one or more data collection nodes and, in response to analyzing the collected performance information, adapting rates at which the data collection nodes send data collection requests to external data sources. Data collection performance information generally may include, but is not limited to, network traffic data, error messages generated by external data sources and/or data collection nodes, computing device performance information, and any other types of information related to a data collection node's ability to collect data from external data sources.

Schedule modification of data collection requests sent to external data sources

The disclosed embodiments provide a system that processes data received from a remote system. During operation, the system sends, from a computer system to a remote system, a request for a local time at the remote system and records a time of transmission of the request. Next, the system obtains, from the remote system, a response to the request, wherein the response includes the local time of the remote system. The system then computes a difference between the time of transmission and the local time of the remote system to determine a time offset that accounts for a time difference between the computer system and the remote system. Finally, the system uses the time offset to standardize timestamps in time-series data received from the remote system, wherein standardizing the timestamps associated with the time-series data comprises adjusting the timestamps to conform to a time standard.

Automatically adjusting timestamps from remote systems based on time zone differences

Adaptive control of data collection requests sent to external data sources

Growing community of inventors

Nanjing, China

Junqing Hao