International Business Machines Corporation

Teradata Us, Inc.

Splunk Inc.

David Everett Simmen

Mir Hamid Pirahesh

Guy Maring Lohman

Ting Yu Leung

Eugene Jon Shekita

Monica Sachiye Urata

Tuong Chanh Truong

Roberta Jo Cochrane

Richard Sefton Sidle

Calisto Paul Zuzarte

George Lapis

Vijayshankar Raman

Steve Yu Zhang

Bruce Gilbert Lindsay

Ledion Bitincka

Vishal Patel

Yunyao Li

Frederick Ralph Reiss

Anjali Betawadkar-Norwood

Markos Zaharioudakis

Hamid Pirahesh

Suresh Thalamati

Marki Volker

Nimrod Megiddo

Christopher Madden Pride

Wei Fan

Peter Haas

Alexander Douglas James

Kevin Scott Beyer

David Marquardt

Timothy Ray Malkemus

Yang Sun

Igor Stojanovski

Thomas Allan Haggie

Michael Jeffrey Winer

Clint Sharp

Volker Gerhard Markl

Jun Rao

Gary Valentin

Daniel Costante Zilio

Hailun Yan

Murugan Kandaswamy

Jeffrey Davis

Armageddon Rhabdizo Brown

Kaarel Truuvert

Marc Andre Chene

Amrittpal Singh Bath

Pratiksha Shah

Michelle Meichiou Jou

Ling Xiao Wang

Shant Hovsepian

Julia Stoyanovich

Esguerra Ma Kharisma

Milind Joshi

Lori G Strain

Kit Man Cheung

Samuel S Lightstone

Ashutosh Vir Vikram Singh

Johann Christoph Freytag

Derrick Poo-Ray Kondo

Tongxin Bai

Aditi Subodh Pandit

Sanjai Tiwari

The disclosed embodiments include a method performed by a data intake and query system. The method includes ingesting each metric including at least one key value and a measured value taken of a computing resource, and storing each metric in an index of a metrics store, where the index defines at least one dimension populated with the at least one key value and a measure populated with the measured value. The method further includes cataloging metadata in a metrics catalog, where the metadata is related to the metrics stored in the metrics store, performing an analysis of metrics data included in the metrics store and/or the metrics catalog to obtain results, and causing display of the results or an indication of the results on a display device.

Metrics store system

The disclosed embodiments include a method performed by a data intake and query system. The method includes receiving a real-time search query including search criteria, and receiving a stream of metrics, where each metric includes a measured value taken of a computing device. The method further includes filtering the metrics to obtain filtered metrics satisfying the search criteria, creating an in-memory summarization data structure based on the filtered metrics, communicating the summarization data to a search head, and providing search results including the summarization data, where the summarization data or data indicative of the summarization data is displayed on a display of a display device.

Real-time search techniques

A system and method for determining optimal query plans within distributed database system employing table operators for performing analytic operations for storing and processing multi-structured data. The optimization of a query plan proceeds through a collaborative exchange between a database system optimizer, or planner, and a table operator, wherein multiple communications between said optimizer and said table operator are conducted to exchange input and output information relevant to optimizing execution of the query and table operator.

Collaborative planning for accelerating analytic queries

A system may include an array of storage devices configured to store a data. The system may further include an array of processing nodes in communication with the array of storage devices. The array of processing nodes may receive a request to perform at least one task associated with the data. The request may include a function call to a function configured to operate on a first data table and a second data table included in the data. The array of processing nodes may partition the first data table among respective subsets of the processing nodes based on a partition key. The array of processing nodes may distribute the second data table among the partitions based on the partition key. The array of processing nodes may execute the function on the first data table and the second data table at each of the partitions. A method and computer-readable medium may also be implemented.

Multi-input SQL-MR

One or more transformation functions can be used in connection or together with one or more compression/decompression techniques. A transformation function can transform data (e.g., a data object) into a form more suitable for compression and/or decompression. As a result, data can be compressed and/or decompressed more effectively. In addition, multiple data objects can be associated with various transformation functions and/or compression/decompression techniques. As a result, different approaches can be taken with respect to compression and decompression of data objects in an effort to find an optimum approach for compression of data objects that may vary significantly from each other and change over time. It will be appreciated that the objects can be associated with transformation functions in a dynamic manner to accommodate changes to data. Also, an extendible and/or extensible system can allow for growth and adaption of new data in forms not currently present or expected.

Transformation functions for compression and decompression of data in computing environments and systems

A data mashup system having information extraction capabilities for receiving multiple streams of textual data, at least one of which contains unstructured textual data. A repository stores annotators that describe how to analyze the streams of textual data for specified unstructured data components. The annotators are applied to the data streams to identify and extract the specified data components according to the annotators. The extracted data components are tagged to generate structured data components and the specified unstructured data components in the input data streams are replaced with the tagged data components. The system then combines the tagged data from the multiple streams to form a mashup output data stream.

Extensible system and method for information extraction in a data processing system

Methods and apparatus, including computer program products, implementing and using techniques for processing a federated query in a federated database system. A federated query is received at a federated database server. A federated query execution plan is generated based on the received federated query. The federated query execution plan defines one or more source servers of the federated database and a unique subquery to be executed on each of the source servers. The subqueries are distributed to the source servers in accordance with the federated query execution plan. The respective subqueries are executed asynchronously at the source servers. The subquery results are passed to a first designated source server defined in the federated query execution plan. The subquery results are joined and aggregated at the first designated source server into a final query result. The final query result is returned to the federated database server.

Efficient processing of queries in federated database systems

A technique for optimizing execution of a query that accesses data stored on a data store connected to a computer. Statistics on one or more pre-defined queries are used to determine an optimal query execution plan for the query. In particular, improved cardinality estimates are generated for one or more query execution plans for the query using statistics of one or more pre-defined queries that vertically overlap the query. These cardinality estimates are used to make more accurate cost estimates, thus improving the likelihood of determining the optimal query execution plan.

Query optimization technique for obtaining improved cardinality estimates using statistics on pre-defined queries

Methods and systems for improving a data transformation operation that converts a source data instance containing repeating elements into a target data instance having a user-specified structure, based solely on a user's specification of a target template. The methods and systems derive and calculate sub-iteration contexts by applying a selected heuristic to the source data instance and the target template, and use these sub-iteration contexts to create a target data instance having a repeating structure that agrees with the user-specified target template. The methods and systems can be customized by the selection of heuristic, and by the specification of explicit sub-iteration contexts that may override the derived contexts.

System and method for the derivation and application of sub-iteration contexts in a transformation operation in a data integration system

A method and system for automatically and adaptively determining query execution plans for parametric queries. A first classifier trained by an initial set of training points is generated. A query workload and/or database statistics are dynamically updated. A new set of training points is collected off-line. Using the new set of training points, the first classifier is modified into a second classifier. A database query is received at a runtime subsequent to the off-line phase. The query includes predicates having parameter markers bound to actual values. The predicates are associated with selectivities. A mapping of the selectivities into a plan determines the query execution plan. The determined query execution plan is included in an augmented set of training points, where the augmented set includes the initial set and the new set.

Automatically and adaptively determining execution plans for queries with parameter markers

A technique for optimizing execution of a query that accesses data stored on a data store connected to a computer. Statistics on one or more automatic summary tables are used to determine an optimal query execution plan for the query. In particular, improved cardinality estimates are generated for one or more query execution plans for the query using statistics of one or more automatic summary tables that vertically overlap the query. These cardinality estimates are used to make more accurate cost estimates, thus improving the likelihood of determining the optimal query execution plan.

Query optimization technique for obtaining improved cardinality estimates using statistics on automatic summary tables

Data in materialized query tables (MQTs) are used as statistics for determining the optimal execution plan for a query. When an MQT is defined, it is examined to determine whether its data provides statistics for determining an optimal execution plan for a query. If so, then the MQT is identified, in the RDBMS, as a source for statistics. Information needed to exploit the MQT data as statistics is cataloged in the RDBMS. This information includes a characterization of the type of statistics provided by the MQT, the table and column distributions represented by those statistics, and a query for later retrieving relevant data from the MQT during the query optimization process. When a query is accepted for execution, the cataloged relevant information about MQTs is examined to determine whether an MQT exists that provides statistics relevant to optimization of the query. If such an MQT exists, then the relevant data is retrieved from the MQT using the cataloged query. Using the retrieved statistics, an optimal execution plan may be determined for the query.

Using data in materialized query tables as a source for query optimization statistics

A method, system, and computer program product to make query processing more robust in the face of optimization errors. The invention validates the statistics and assumptions used for compiling a query as the query is executed and, when necessary, progressively re-optimizes the query in mid-execution based on the knowledge learned during its partial execution. The invention selectively places a number of CHECK operators in a query execution plan to validate the optimizer's cardinality estimates against actual cardinalities. Errors beyond a threshold trigger re-optimization, and the optimizer decides whether the old plan is still optimal and whether to re-use previously computed results. The invention addresses arbitrary SQL queries whose plans can contain sub-queries, updates, trigger checking, and view maintenance operations. The invention can handle concurrent update transactions or updates of common sub-expressions in a query execution plan without compromising consistency and isolation as locking information is tied to the record ID.

System, method, and computer program product for progressive query processing

Growing community of inventors

San Jose, CA, United States of America

David Everett Simmen