Databricks Inc.

Amazon Technologies, Inc.

Frederick Ryan Johnson

Prakhar Jain

Bart Samwel

Anurag Windlass Gupta

Andrew Edward Caldwell

Bhaven Avalani

Ippokratis Pandis

Naresh Chainani

Michail Petropoulos

Martin Grund

Srividhya Srinivasan

Naga Raju Bhanoori

Vijayan Prabhakaran

Venu Gopal Nayar

Terry Kim

Hebatalla Mohamed Mohamed Aly Eldakiky

Efficient superblock data structures are implemented for a data set. A superblock data structure, a log to commit changes to a superblock data structure, and an auxiliary data structure may be updated when an update to a data set is performed. The update may update respective location headers in the auxiliary data structure and the superblock data structure.

Efficient superblock data structures for database systems

A data processing service generates a data classifier tree for managing data files of a data table. The data classifier tree may be configured as a KD-classifier tree and includes a plurality of nodes and edges. A node of the data classifier tree may represent a splitting condition with respect to key-values for a respective key. A node of the data classifier tree may be associated with one or more data files assigned to the node. The data files assigned to the node each include a subset of records having key-values that satisfy the conditions represented by the node and parent nodes of the node. The data processing service may efficiently cluster the data in the data table while reducing the number of data files that are rewritten when data is modified or added to the data table.

Data file clustering with KD-classifier trees

A data processing service uses minor compactions for committing transactions to a data table. The service may receive requests to commit transactions to a data table and write metadata for the transactions to log files, and generate a checkpoint file aggregating the transactions described in the log files to compute a data table state at a first time. The service may receive requests to commit a set of transactions and write metadata for the set of transactions to a set of log files. The service may determine that a number of log files in the set of log files reaches a threshold commit number, generate a minor compaction file aggregating the set of transactions, and generate a second checkpoint file aggregating the data table state at the first time with information from the minor compaction file to compute the data table state at a second time.

Generating minor compactions to capture aggregated actions for commit ranges to data files

A data tree for managing data files of a data table and performing one or more transaction operations to the data table is described. The data tree is configured as a KD-epsilon tree and includes a plurality of nodes and edges. A node of the data tree may represent a splitting condition with respect to key-values for a respective key. A leaf node of the data tree may correspond to a data file for a data table that includes a subset of records having key-values that satisfy the condition for the node and conditions associated with parent nodes of the node. A parent node may correspond to a file including a buffer that stores changes to data files reachable by this parent node, and also includes dedicated storage to pointers of the child nodes. By using the data tree, the data processing system may efficiently cluster the data in the data table while reducing the number of data files that are rewritten.

Data file clustering with KD-epsilon trees

Data ingestion using data file clustering with KD-epsilon trees

Burst performance of a database query may be determined according to a size of the database query. A query to a database may be received. A size may be determined for the query. If the size is less than a size threshold assigned to a first query engine, then the query may be performed at the first query engine. If the size is greater than or equal to the size threshold assigned to the first query engine, then the query may be performed at a second query engine.

Burst performance of database queries according to query size

Growing community of inventors

Orem, UT, United States of America

Frederick Ryan Johnson