Amazon Technologies, Inc.

Hewlett-packard Development Company, L.p.

Other

Amiato, Inc.

Benjamin Albert Sowell

Mehul A Shah

Stavros Harizopoulos

Anurag Windlass Gupta

Bohou Li

Nathan A Binkert

Douglas Brian Terry

George Steven McPherson

Vijayan Prabhakaran

Kevin R Meyer

Dimitris Tsirogiannis

Bryan D Kaplan

Nathan Lorenzo Binkert

Gopinath Duddi

Prajakta Datta Damle

Dimitrios Tsirogiannis

Jakub Kulesza

Anthony A Virtuoso

Indrajit Roy

Wojciech Golab

Carlos Garcia-Alvarado

Supratik Chakraborty

Zhuzeng Lu

James Thomas Kiraly

Linan Zheng

Wojclech Golab

Vinay Agrawal

Vikas Malik

Tushar Poddar

Mukesh Punhani

Shashank Bhardwaj

Roman Gavrilov

Brian Scott Ross

A protocol for implementing ACID transactions that provides snapshot isolation in a distributed setting that does not require synchronized clocks is described. The protocol ensures at commit time that transactions touching common objects do not commit out of order. The protocol can be used in the context of a distributed data lake built on an object store in which clients can transactionally add or remove objects from logical tables.

Transactional protocol for snapshot isolation without synchronized clocks

Data transformation workflows may be generated to transform data objects. A source data schema for a data object and a target data format or target data schema for a data object may be identified. A comparison of the source data schema and the target data format or schema may be made to determine what transformations can be performed to transform the data object into the target data format or schema. Code to execute the transformation operations may then be generated. The code may be stored for subsequent modification or execution.

Generating data transformation workflows

Techniques for storage optimization in a distributed object store are described. A storage optimization service of a provider network monitors changes to data objects in a distributed object store that are part of a data lake and are referenced by a table index. The storage optimization service determines whether particular storage optimizations involving the data objects would be beneficial, prioritizes the ordering of these optimizations with a focus on performing impactful optimizations first, while intelligently scheduling the optimizations to avoid overutilization of available resources.

Storage optimization in a distributed object store

Techniques for using tree data structures to maintain a transactionally consistent set with support for time-travel queries are described. When a transaction commits, a new version of the tree data structure is created using a copy-on-write based method such that the tree shares internal nodes with previous trees to save space. This approach may be used in the implementation of a transactional data catalog in which the files that make up a table are stored in a transactional set.

Tree-based approach for transactionally consistent version sets

Techniques for implementing systems using transactional version sets are described. Transactional version sets or t-sets include a collection of elements, each having a collection of metadata. A t-set is transactional in that a sequence of updates to one or more t-sets are made within an atomic transaction. A t-set is versioned since each committed transaction that updates it produces a new timestamped version that can be accessed via time-travel queries.

Transactional version sets

A data transformation system includes a schema inference module and an export module. The schema inference module is configured to dynamically create a cumulative schema for objects retrieved from a first data source. Each of the retrieved objects includes (i) data and (ii) metadata describing the data. Dynamically creating the cumulative schema includes, for each object of the retrieved objects, (i) inferring a schema from the object and (ii) selectively updating the cumulative schema to describe the object according to the inferred schema. The export module is configured to output the data of the retrieved objects to a data destination system according to the cumulative schema.

Creation of a cumulative schema based on an inferred schema and statistics

Version controlled Extract, Transform, Load (ETL) code may be hosted for developing or executing the ETL job in an ETL system. A version of ETL code may be obtained from version controlled code store and maintained in a data store. Development or execution clients may submit access requests for the version of ETL code which may be serviced from the version stored in the data store. Updates to the version of the ETL code may be eventually committed to the version controlled code store. The latest version of ETL code may also be obtained from the version controlled code store when providing the ETL code in response to a request to retrieve the ETL code.

Hosting version controlled extract, transform, load (ETL) code

A first column of a first database table may be transferred to a second table in a combined truncate and append operation in a transaction that may be atomic and reversible. References data for the first column may be removed from the first table and added to the second table. The combined operation may be reversed by removing, from the second table, references to blocks of data added to the second table prior to commencing the combined operation. A second column corresponding on a row-by-row basis to the first column may be added to the second table, where the data is indicative of visibility of data in the first column to other transactions on the second table.

Truncate and append database operation

A method of operating a data analysis system includes retrieving objects from a data source. Each of the retrieved objects includes (i) data and (ii) metadata describing the data. The method further includes dynamically creating a cumulative schema by, for each object of the retrieved objects: (i) inferring a schema from the object based on the metadata of the object and inferred data types of elements of the data of the object, (ii) creating a unified schema, wherein the unified schema describes both (a) the object described by the inferred schema and (b) a cumulative set of objects described by the cumulative schema, and (iii) storing the unified schema as the cumulative schema. The method further includes exporting the data of each of the retrieved objects to a data warehouse.

Scalable analysis platform for semi-structured data

A balanced distribution of sort order values may be implemented for a multi-column sort order of a database table. Columns of the database table to be included in the multi-column sort order may be identified. Some columns containing string data values may be converted to equally-sized integer data values. The data values of columns may be evaluated to determine buckets representing the ranges of data values within the columns for depth-balanced histograms of the columns. Multi-column sort order values may be generated for individual entries in the database table according to bucket values assigned to the buckets that include the columns values of the individual entries. The entries of the database table may then be stored according to a sorted ordering of multi-column sort order values for the entries.

Balanced distribution of sort order values for a multi-column sort order of a relational database

A method of operating a query system includes retrieving objects from a data source. Each of the retrieved objects includes (i) data and (ii) metadata describing the data. The method further includes dynamically creating a cumulative schema. The dynamically creating includes, for each object of the retrieved objects, (i) inferring a schema from the object based on the metadata of the object and inferred data types of elements of the data of the object, (ii) creating a unified schema, and (iii) storing the unified schema as the cumulative schema. The unified schema describes both (a) the object described by the inferred schema and (b) a cumulative set of objects described by the cumulative schema. The method further includes storing the data of each of the retrieved objects in a storage service.

A distributed balanced tree having a plurality of nodes distributed across a plurality of servers is accessed. Version information associated with a node of the distributed balanced tree is retrieved. Validity of a lookup transaction performed on the balanced tree is determined by verifying a version value of only the leaf node accessed during the lookup operation against the retrieved version information.

Validation of distributed balanced trees

A method for managing data using writeable snapshots in a multi-versioned, distributed B-tree comprising nodes distributed over a plurality of servers, includes receiving a transaction request specifying a transaction type, the transaction having a transaction snapshot id, and determining the transaction type. If the transaction type is an update transaction, the method includes determining a node to update, which is defined by a node snapshot id, and if the transaction snapshot id is greater that the node snapshot id, the method copies the node and updates the copy to create a writeable snapshot, assigns the transaction snapshot id as a node snapshot id of the copy, and places a pointer from the node to the copy.

Growing community of inventors

Mountain View, CA, United States of America

Benjamin Albert Sowell