Amazon Technologies, Inc.

Ippokratis Pandis

Naresh Chainani

Martin Grund

Mengchu Cai

Eric Ray Hotinger

Michail Petropoulos

William Michael McCreedy

Fabian Oliver Nagel

Gaurav Saxena

Pavel Sokolov

Anurag Windlass Gupta

Gokul Soundararajan

Yannis Papakonstantinou

Mohammad Rezaur Rahman

Mohammad Foyzur Rahman

Davide Pagano

Vivek Ramamoorthy

Jingyi Qing

Bruce William McGaughy

Sanjay Wangoo

Andrew Edward Caldwell

Bhaven Avalani

Krishna Chaitanya Gudipati

Venkatraman Govindaraju

Mingda Li

Sanuj Basu

Kiran Kumar Chinta

Gopal Paliwal

Neeraja Rentachintala

Dengfeng Li

Orestis Polychroniou

Matthew James Perron

Naveen Muralimanohar

Chao Duan

Vadim Skipin

Yan Valerie Leshinsky

Saileshwar Krishnamurthy

Nathan Lorenzo Binkert

Kamal Kant Gupta

Maximiliano Maccanti

Justin Jon Levandoski

Yongsik Yoon

Sangyong Hwang

Balakrishnan Narayanaswamy

Punit Rajgaria

Murali Brahmadesam

Hemanth Satyanarayana

Frederick Ryan Johnson

Matthew Perry Abrams

Induja Sreekanthan

Srividhya Srinivasan

Naga Raju Bhanoori

Chunbin Lin

Aditya Subrahmanyan

Panagiotis Parchas

Todd Jeffrey Green

Christos Stavrakakis

Arvind Jayasundar

Nemanja Boric

Vladimir Ponomarenko

Yousuf Hussain Syed Mohammad

Mikhail Krestyaninov

Zhixing Ma

Luxiang Yin

Pulkit Jagdishchandra Bhavsar

Sebastian Hillig

Chenqin Xu

Maor Kleider

Wenchuan An

Hanna Loboda

Hui Shi

Demala Palliye Gurunnanselage Shermal Ruwantha Fernando

Qichen Zhang

Nikolaos Armenatzoglou

Sushim Mitra

John Thomas

Monish Gupta

Weihang Song

Bin Zhang

Mengyang Li

Hitenkumar Sonani

Kathiravan Kalimuthu

Ankil Shah

Michalis Petropoulos

Yao Xiao

Siyi Zhang

Yehan Zhang

Vaishali Ravindra Narkhede

Johannes Wust

A system for delegation of authorization management of a shared database of a database service is described. The database service includes a control plane and a producer database engine. The producer database engine receives a creation request to create a datashare for a database. The creation request delegates, to a data exchange service, authorization management to access the datashare. The control plane is configured to update permission data for the datashare to indicate that authorization to the datashare is managed by the data exchange service. The producer database engine is further configured to receive a request, from a consumer database engine, to obtain metadata used to perform a query to the database of the datashare, determine that the data exchange service authorized the consumer database engine to access the datashare based on the permission data for the datashare, and return the metadata to the consumer database engine.

Delegated authorization for consumers of shared databases

Database data of a database service may be privately shared across provider network regions. A database producer can authorize an account of to access a database in another region of the provider network. The consumer of the database can associate the database in order to accept the permission to access the database. The database can then be created as an external table at a consumer database engine and metadata for the database privately shared between producer and consumer database engines that can be used to query the external table.

Privately sharing database data across provider network regions

Dynamic selection of topologies for database data may be performed for performing queries. Different possible query plans for different assignments of topologies are determined for a query. One of the query plans is selected according to a cost-based model that assigns costs to different uses of the different topologies. The query is then performed using the selected query plan.

Dynamic selection of database data topologies for performing queries

Methods for replicating transactional tables of a transactional database to an analytical database and maintaining updates to those transactional table representations are disclosed. Snapshots of the transactional tables are provided to the analytical database via a transport mechanism, such as a data storage service or a data streaming service, and stored at the analytical database. Then, checkpoints comprising portions of a change-data-capture log that has recorded transactional changes to the transactional tables of the transactional database are provided to the analytical database via the same or different transport mechanism and used to commit those transactional changes to the snapshot representations. The snapshot representations may be used to respond to incoming analytical queries in order to provide real-time querying results.

Hybrid transactional and analytical processing architecture for optimization of real-time analytical querying

Idle periods may be for management actions at processing clusters for managed databases. A leader node of a processing cluster for a managed database may monitor a network endpoint at a proxy service associated with a database managed by the database service. An idle period for the database may be detected. A management action for the processing cluster may be determined to be performed during the detected idle period. The leader node may cause the determined management action to be performed.

Detecting idle periods at network endpoints for management actions at processing clusters for managed databases

When a query is received by a stateful data processing service, the service determines, for each table scan (and associated operations) of a query, whether to select the table scan for execution by a stateless data processing service. The selected table scans are sent to the stateless data processing service for execution, and results are received by the stateful data processing service. The stateful data processing service may also execute other table scans of the query locally, against a local data cache. If the data is not present in the local data cache, then the stateful data processing service will copy the table data into the local data cache before executing the table scan. A query result based on the remote and/or local table scans may then be returned to the client.

Selecting between hydration-based scanning and stateless scale-out scanning to improve query performance

Database query processing may be scaled using additional processing clusters. A database query is received at a processing cluster. A determination is made as to whether additional processing clusters will be used to process the database query. Operations to cause compute nodes of the processing cluster to instruct operations at the additional processing clusters are included in a plan generated to perform database queries determined to use additional processing clusters. The plan is executed to be perform the database query causing compute nodes of the processing cluster to send instructions to corresponding additional processing clusters in order to generate and return a response to the database query.

Scaling database query processing using additional processing clusters

Dynamic plan interpretation may be implemented to perform queries. A query is received at a database system. A plan to perform the query is generated. For at least a portion of the plan to perform the query, a domain-specific language may be applied to interpret and execute the portion of the plan to perform the query using one or more previously compiled executors instead of generating and compiling code to execute the portion of the plan. Returning a result of the query according to the execution of the plan.

Dynamic selection of plan interpretation to perform queries

Burst performance of a database query may be determined according to a size of the database query. A query to a database may be received. A size may be determined for the query. If the size is less than a size threshold assigned to a first query engine, then the query may be performed at the first query engine. If the size is greater than or equal to the size threshold assigned to the first query engine, then the query may be performed at a second query engine.

Burst performance of database queries according to query size

Methods for implementing insert, update, and delete events of a change-data-capture log in transactional order to a representation of a transactional table are disclosed. Insert events may be applied and committed to the representation, while delete events may be buffered and committed to a shadow table. Such buffering may allow the delete events to be applied and committed as a batch to the given representation as a later background operation, allowing for an optimized application of the change-data-capture log to the representation. Once the delete events are committed to the representation, they are removed from the shadow table. Update events may be similarly treated in which the insert event component of the update event is applied and committed to the representation while the delete event component is committed to the shadow table, before being later committed to the representation and removed from the shadow table.

Optimization of application of transactional information for a hybrid transactional and analytical processing architecture

Authorization is divided between a control plane and a data plane for sharing database data. A producer database engine can create a shared database via a data plane interface. A producer can then authorize access to the shared database via a control plane interface to a consumer. A consumer can associate the authorization granted to the consumer with a consumer database engine via the control plane interface.

Dividing authorization between a control plane and a data plane for sharing database data

A primary query engine may receive a query and determine whether the query is eligible for performance at a secondary query engine. If eligible, the primary query engine may evaluate the availability of the first query engine to perform the query. The first query engine may determine whether to assign the query to the primary query engine or to the secondary query according to availability evaluation. For queries assigned to the secondary query engine, the primary query engine may send a request to the secondary query engine to being processing of the query.

Dynamically assigning queries to secondary query processing resources

Queries to a consistent view of a data set across different types of query engines are performed. Portions of a query received at a first type of query engine that can be performed at a different type of a query engine can be identified. A consistent view of a data set stored in a common data store can be provided by the first type of query engine to the second query engine. When the different type of query engine performs the portion of the query, the different type of query engine applies the consistent view of the data set. A result for the query can be returned by the first type of query engine based on results from the different type of query engine.

Performing queries to a consistent view of a data set across query engine types

Compiled portions of code generated to perform a query plan at a query engine may be shared with other query engines. A data store, separate from the query engines, may store compiled portions of query code generated for different queries. If a query engine does not have a locally stored compiled portion of query code, then the separate data store may be accessed in order to obtain a compiled portion of query code, allowing reuse of compiled query code across different queries engines for queries directed to different databases.

Sharing compiled code for executing queries across query engines

Online restore may be performed between databases with different topologies while applying a custom data distribution. A request to restore a database into a different topology of nodes may be received. A plan to move different portions of the database from a current topology to the new topology made using a general distribution scheme. The plan may be performed to move the different portions of the database into the new topology and the database made available for access using the new topology. A background process may be applied to modify the distribution of the database at the new topology to match a custom distribution scheme that was implemented at the current topology.

Online restore to different topologies with custom data distribution

Growing community of inventors

Menlo Park, CA, United States of America

Ippokratis Pandis