International Business Machines Corporation

Amazon Technologies, Inc.

Databricks Inc.

Vuk Ercegovac

Yannis Papakonstantinou

Peter Haas

Gaurav Saxena

Balakrishnan Narayanaswamy

Enrico Siragusa

Kevin Scott Beyer

Andrey Balmin

Andre Hernich

Eugene Jon Shekita

Fei Xu

Alfredo Alba

Gokul Soundararajan

Thomas D Griffin

Chad Eric DeLuca

Sriram Krishnamurthy

Jun Rao

John Sismanis

George Constantin Caragea

Carl-Christian Kanne

Nikolaos Koulouris

Jiayuan Chen

Liping Peng

Asim V Singh

Kevin B Wang

Fatma Ozcan

Michael Paul Armbrust

Christos Faloutsos

Marcel Kutsch

Mukul Murthy

Nitin Sharma

Panagiotis Parchas

Mario Guerriero

Paul Lappas

Xi Liang

Joseph Torres

Min Yang

Jesse E Jackson

Konstantinos Karanasos

Chunyang Xia

Rares Vernica

A system receives ETL specification for processing stream data, including a transform operation represented using a database query specification. The system generates a dataflow graph of a sequence of database queries by decomposing the database query into a first database query that generates an intermediate results table, and a second database query that receives as input the intermediate results table and outputs data used for performing the transform operation. The system executes the sequence of database queries for performing the transform operation on stream data received from the source. When receiving an incremental data set, the system determines an output change set based on the received incremental data set by traversing an execution plan and processing each operator in the execution plan, and computing a change set of a particular operator from the change sets output by the one or more other operators based on the incremental data set.

Incremental execution of extract, transform, load process using microtechniques architecture

Queries that reference materialized views may be rewritten to compensate for stale materialized views. A query may be received that references a materialized view. A determination may be made as to whether the materialized view is stale. For a stale materialized view, the query may be rewritten to generate a query plan that obtains changes made to base tables for the materialized view not included in the stale materialized view and considers the change(s) when generating a result for the query from the materialized view. The rewritten query may then be performed to provide a result as if the materialized view were up-to-date.

Rewriting queries to compensate for stale materialized views

Stored procedures are generated to perform incremental updates to a materialized view for a database. When a request to create a materialized view is received, one or more internal tables are created from the database. A stored procedure is generated that when executed will update the materialized view of the database. The stored procedure may obtain changes to the database that were not included in the internal tables and update the internal tables to include the obtained changes. The stored procedure may be performed automatically and in response to requests.

Stored procedures for incremental updates to internal tables for materialized views

A database system may use a machine learning model creation system to create a machine learning model from data stored in the database system responsive to a request from a client. The database system may obtain an executable version of the machine learning model, based on an uncompiled hardware agnostic version of the machine learning model, according to the hardware configuration of one or more computing resources selected by the database system to perform requests to the database system that invoke the machine learning model to generate predictions.

Dynamic compilation of machine learning models based on hardware configurations

A database system may include a machine learning model which may be used to perform various data analytics for data stored in the database system. In response to a request to invoke the machine learning model to generate a prediction from data stored in the database system, the database system may perform one or more optimization operations, as part of a query plan, to prepare the data to make it suitable for use by the machine learning model.

Integrating query optimization with machine learning model prediction

Materialized views for a database system may be automatically refreshed according to performance benefits. Materialized views may be ordered according to determined performance benefits for the materialized views indicating the performance benefit obtained when a materialized view is used to perform a query at the database system. Materialized views may be selected for refresh operations according to the ordering based on a capacity of the database system to perform refresh operations.

Automatically refreshing materialized views according to performance benefit

Match rules for rewriting queries to use materialized views may be dynamically generated by a database system. A database system may generate rules that indicate whether a given query can use a materialized view and how to rewrite the given query to use the materialized view. A query may be received and the rules may be applied to the query to determine that the query can use the materialized view and to rewrite the query to use the materialized view. The rewritten query can then be executed.

Dynamic generation of match rules for rewriting queries to use materialized views

Algebraic operations are applied to change result sets for source tables to update a materialized view of a database incrementally. A procedure is determined for updating a materialized view that generates result sets for changes to source tables for a materialized view. The procedure performs one or more operations to update the materialized view using the result sets as operands in a relational algebra operator in the procedure to include the changes in the result sets in the materialized view.

Applying relational algebraic operations to change result sets of source tables to update a materialized view

Databases may enforce foreign key constraints for efficient materialized view updates. A materialized view may be created for a database. Updates to the database may be received and performed according to a foreign key constraint enforced with respect to the updates. An update to the materialized view may be performed to make the materialized view consistent with the updates to the database. Performance of a join operation to update the materialized view that would return a result violating the foreign key constraint may be excluded from the update to the materialized view.

Enforcing foreign key constraints for efficient materialized view updates

Systems and associated methods for highly parallel processing of parameterized simulations are described. Embodiments permit processing of stochastic data-intensive simulations in a highly parallel fashion in order to distribute the intensive workload. Embodiments utilize methods of seeding records in a database with a source of pseudo-random numbers, such as a compressed seed for a pseudo-random number generator, such that seeded records may be processed independently in a highly parallel fashion. Thus, embodiments provide systems and associated methods facilitating quicker data-intensive simulation by enabling highly parallel asynchronous simulations.

Systems and methods for highly parallel processing of parameterized simulations

In one embodiment, a computer-implemented method includes selecting one or more sub-expressions of a query during compile time. One or more pilot runs are performed by one or more computer processors. The one or more pilot runs include a pilot run associated with each of one or more of the selected sub-expressions, and each pilot run includes at least partial execution of the associated selected sub-expression. The pilot runs are performed during execution time. Statistics are collected on the one or more pilot runs during performance of the one or more pilot runs. The query is optimized based at least in part on the statistics collected during the one or more pilot runs, where the optimization includes basing cardinality and cost estimates on the statistics collected during the pilot runs.

Dynamic query optimization with pilot runs

Stratified sampling of a plurality of records is performed. A plurality of records are partitioned into a plurality of splits, wherein each split includes at least a portion of the plurality of records. The split of the plurality of splits is provided to a mapper. The mapper assigns at least a portion the records of the at least one split to a group based on a strata of the assigned records, and filters the records of the group based on a comparison of the weights of the records to a local threshold of the mapper. The mapper updates the local threshold of the mapper by communicating with a coordinator. The mapper shuffles the group to a reducer, where the reducer filters the records of the group based on the weights of the records. The reducer provides a stratified sampling of the plurality of records based on the group.

Stratified sampling using adaptive parallel data processing

Growing community of inventors

Campbell, CA, United States of America

Vuk Ercegovac