Amazon Technologies, Inc.

Pivotal Software, Inc.

Michail Petropoulos

George Constantin Caragea

Amr El-Helw

Anurag Windlass Gupta

Ippokratis Pandis

Mohamed F Soliman

Andrew Edward Caldwell

Florian Michael Waas

Lyublena Rosenova Antova

Zhongxian Gu

Carlos Garcia-Alvarado

Mohammad Foyzur Rahman

Stavros Harizopoulos

Martin Grund

Venkatesh Raghavan

Panagiotis Parchas

Andrea Olgiati

Saileshwar Krishnamurthy

Nathan Lorenzo Binkert

Yannis Papakonstantinou

Bhaven Avalani

Kamal Kant Gupta

Mengchu Cai

Naresh Chainani

Justin Jon Levandoski

John Benjamin Tobler

Yongsik Yoon

Sangyong Hwang

Christos Faloutsos

Frederick Ryan Johnson

Fabian Oliver Nagel

Srividhya Srinivasan

Ramakrishna Kotla

Maor Kleider

Mohamed Y Soliman

Burst performance of a database query may be determined according to a size of the database query. A query to a database may be received. A size may be determined for the query. If the size is less than a size threshold assigned to a first query engine, then the query may be performed at the first query engine. If the size is greater than or equal to the size threshold assigned to the first query engine, then the query may be performed at a second query engine.

Burst performance of database queries according to query size

Queries to a consistent view of a data set across different types of query engines are performed. Portions of a query received at a first type of query engine that can be performed at a different type of a query engine can be identified. A consistent view of a data set stored in a common data store can be provided by the first type of query engine to the second query engine. When the different type of query engine performs the portion of the query, the different type of query engine applies the consistent view of the data set. A result for the query can be returned by the first type of query engine based on results from the different type of query engine.

Performing queries to a consistent view of a data set across query engine types

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for precisely tracking memory usage in a multi-process computing environment. One of the methods includes implementing an instance of a memory usage tracker (MUT) in each process running in a node of a computer system. A MUT can maintain an account of memory usage for each of multiple logical owners running on a process on which the MUT is running. The MUT can determine an actual memory quota for each owner, and enforce the actual memory quota of the owner. Enforcing the actual memory quota of the owner can include receiving each memory allocation request, checking each allocation request and a current state of the account against the actual quota, approving or rejecting each allocation request, communicating the approval or rejection to an underlying memory manager, and updating the owner account for each approved allocation request.

Precisely tracking memory usage in multi-process computing environment

A tree-based format may be implemented for data stored in a data store. A table may be maintained across one or multiple storage nodes in storage slabs. Storage slabs may be mapped to different nodes of a tree. Each node of the tree may be assigned a different range of distribution scheme values which identify what portions of the table are stored in the storage slab. Storage slabs mapped to child nodes in the tree may be assigned portions of the range of distribution scheme values assigned to a parent. Storage nodes may be added or removed for storing the table. Storage slabs may be moved from one storage node to another in order to accommodate the addition or removal of storage nodes.

Tree-based format for data storage

Using a graph representation of join history may be performed to distribute database data. Join history may be collected, captured, or tracked which describes the history of join operations between columns of different tables in a database. A graph representation of the join history may be generated. The graph representation may indicate a likelihood of different joins that may be performed between the columns of the tables of a database. An evaluation of the join history may be performed to identify columns for tables in the database to distribute the data of the tables amongst multiple storage locations according to the identified columns.

Using a graph representation of join history to distribute database data

A query that references nested data may have sub-expressions included in a subquery for nested data to perform the query. A first query engine may rewrite the query to include one or more sub-expressions that reference the nested data into a subquery for the nested data as part of generating a plan to perform the query. A second query engine may then be caused to perform the subquery for the nested data. A result of the query may be inclusive of the performance of the subquery at the second query engine.

Combining nested data operations for distributed query processing

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for dynamic partition selection. One of the methods includes receiving a representation of a query plan generated for a query, wherein the query plan includes a dynamic scan operator that represents a first computing node obtaining tuples of one or more partitions of a table from storage and transferring the tuples to a second computing node that executes a parent operator of the dynamic scan operator. A partition selector operator is generated corresponding to the dynamic scan operator. A location in the query plan is determined for the partition selector operator. A modified query plan is generated having the partition selector operator at the determined location.

Dynamic partition selection

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for tracking memory usage. One of the methods includes implementing an instance of a memory usage tracker (MUT) in each process running in a node of a computer system. The MUT initiates a free process for a memory chunk. A current generation of the memory chunk is determined. The MUT determines whether the mapping element's generation precedes the current generation. If the mapping element's generation is old, the MUT treats the long-living rollover account as the owner of the memory chunk to be freed and if the current generation and the mapping element's generation match, then the MUT considers the owner associated with the mapping element as the owner of the chunk. The MUT then adjusts the balance of the determined memory account and/or sub-account and the balance of the mapping element by the freed memory amount.

Cardinality values can be derived from an approximate quantile summary. An approximate quantile summary can be generated for a column of a database table at data ingestion, data update, upon request, in response to a query, and in various other scenarios. When a query is received that includes a predicate directed to the column of the approximate quantile summary, a cardinality value may be derived from the boundary values of one or more quantiles that include the predicate. The cardinality value may then be used to select a query plan. The query may be performed according to the selected query plan.

Deriving cardinality values from approximate quantile summaries

Relocating data sharing operations for query processing may be implemented when generating plans to perform a query. A query operation that causes a node to share data obtained from a different set of nodes with other nodes in a same set of nodes may be identified. The identified query operation may be replaced with another operation that causes the different set of nodes to share the data directly with the other nodes in the same set of nodes.

Relocating data sharing operations for query processing

Queries received at a query engine may be executed for structured data and not-structured data. A query execution plan may be generated for the query that includes stateless operations to apply the query to the not-structured data at remote query processing engines. The remote query processing engines may perform the stateless operations and return results to the query engine. The query engine may generate a result for the query based on the results received from the remote query engine as well as results determined as part of applying the query to structured data. The result to the query may be returned.

Executing queries for structured data and not-structured data

Data stores may implement prioritized scheduling of data store access requests. When new access requests are received, the new access requests may be scheduled for prioritized execution on processing resources. Access requests that are currently being executed with prioritized execution may be reprioritized to make additional capacity for prioritized execution of the new access requests. Prioritized execution may be automatically enabled or disabled for a data store based on monitoring of performance metrics for executing access requests.

Prioritized scheduling of data store access requests

Growing community of inventors

San Francisco, CA, United States of America

Michail Petropoulos