Amazon Technologies, Inc.

Hewlett-packard Development Company, L.p.

Hewlett Packard Enterprise Development LP

Other

Amiato, Inc.

Stavros Harizopoulos

Mehul A Shah

Benjamin Albert Sowell

Nathan A Binkert

Anurag Windlass Gupta

Michail Petropoulos

Dimitris Tsirogiannis

Benjamin A Sowell

Dimitrios Tsirogiannis

Andrew Edward Caldwell

Arif Abdulhusein Merchant

Alkiviadis Simitsis

Kevin R Meyer

Bryan D Kaplan

Goetz Graefe

William Kevin Wilkinson

Nathan Lorenzo Binkert

Janet Lynn Wiener

Harumi Kuno

Mehul Jayant Shah

Andrea Olgiati

John Benjamin Tobler

Indrajit Roy

Christos Faloutsos

Wojciech Golab

Carlos Garcia-Alvarado

Panagiotis Parchas

Themistoklis Palpanas

Zhuzeng Lu

Ramakrishna Kotla

Mohit Saxena

Enrico Iori

A tree-based format may be implemented for data stored in a data store. A table may be maintained across one or multiple storage nodes in storage slabs. Storage slabs may be mapped to different nodes of a tree. Each node of the tree may be assigned a different range of distribution scheme values which identify what portions of the table are stored in the storage slab. Storage slabs mapped to child nodes in the tree may be assigned portions of the range of distribution scheme values assigned to a parent. Storage nodes may be added or removed for storing the table. Storage slabs may be moved from one storage node to another in order to accommodate the addition or removal of storage nodes.

Tree-based format for data storage

Using a graph representation of join history may be performed to distribute database data. Join history may be collected, captured, or tracked which describes the history of join operations between columns of different tables in a database. A graph representation of the join history may be generated. The graph representation may indicate a likelihood of different joins that may be performed between the columns of the tables of a database. An evaluation of the join history may be performed to identify columns for tables in the database to distribute the data of the tables amongst multiple storage locations according to the identified columns.

Using a graph representation of join history to distribute database data

A data transformation system includes a schema inference module and an export module. The schema inference module is configured to dynamically create a cumulative schema for objects retrieved from a first data source. Each of the retrieved objects includes (i) data and (ii) metadata describing the data. Dynamically creating the cumulative schema includes, for each object of the retrieved objects, (i) inferring a schema from the object and (ii) selectively updating the cumulative schema to describe the object according to the inferred schema. The export module is configured to output the data of the retrieved objects to a data destination system according to the cumulative schema.

Creation of a cumulative schema based on an inferred schema and statistics

A first column of a first database table may be transferred to a second table in a combined truncate and append operation in a transaction that may be atomic and reversible. References data for the first column may be removed from the first table and added to the second table. The combined operation may be reversed by removing, from the second table, references to blocks of data added to the second table prior to commencing the combined operation. A second column corresponding on a row-by-row basis to the first column may be added to the second table, where the data is indicative of visibility of data in the first column to other transactions on the second table.

Truncate and append database operation

Data stores may implement prioritized scheduling of data store access requests. When new access requests are received, the new access requests may be scheduled for prioritized execution on processing resources. Access requests that are currently being executed with prioritized execution may be reprioritized to make additional capacity for prioritized execution of the new access requests. Prioritized execution may be automatically enabled or disabled for a data store based on monitoring of performance metrics for executing access requests.

Prioritized scheduling of data store access requests

A method of operating a data analysis system includes retrieving objects from a data source. Each of the retrieved objects includes (i) data and (ii) metadata describing the data. The method further includes dynamically creating a cumulative schema by, for each object of the retrieved objects: (i) inferring a schema from the object based on the metadata of the object and inferred data types of elements of the data of the object, (ii) creating a unified schema, wherein the unified schema describes both (a) the object described by the inferred schema and (b) a cumulative set of objects described by the cumulative schema, and (iii) storing the unified schema as the cumulative schema. The method further includes exporting the data of each of the retrieved objects to a data warehouse.

Scalable analysis platform for semi-structured data

A method of operating a query system includes retrieving objects from a data source. Each of the retrieved objects includes (i) data and (ii) metadata describing the data. The method further includes dynamically creating a cumulative schema. The dynamically creating includes, for each object of the retrieved objects, (i) inferring a schema from the object based on the metadata of the object and inferred data types of elements of the data of the object, (ii) creating a unified schema, and (iii) storing the unified schema as the cumulative schema. The unified schema describes both (a) the object described by the inferred schema and (b) a cumulative set of objects described by the cumulative schema. The method further includes storing the data of each of the retrieved objects in a storage service.

An embodiment provides a system and method for transaction commitment and replication. The method includes receiving a minitransaction from a client node at one or more memory nodes, wherein each memory node includes a number of replicas. The minitransaction is a type of transaction which atomically executes any combination of reading, comparing, and writing to any of a number of memory locations. The method also includes determining, for a leader of the replicas within a memory node, whether the leader is able to commit the minitransaction and stabilizing state changes of the minitransaction within a transaction log using a consensus procedure to update the replicas. The method further includes committing the minitransaction if, at each memory node, a quorum of the replicas is able to stabilize the minitransaction, or aborting the minitransaction otherwise.

Transaction commitment and replication in a storage system

A shiftable memory supporting bimodal data storage includes a memory having built-in shifting capability to shift a contiguous subset of data stored in the memory from a first location to a second location within the memory. The shiftable memory further includes a bimodal data storage operator to operate on a data structure comprising the contiguous subset of data words and to provide in-place insertion of a data value using the built-in shifting capability.

Shiftable memory supporting bimodal storage

A computer cluster with objectives-based resource sharing. The cluster includes cloud nodes each with one or more resources, a terminal, data storage, and an allocation node to monitor cloud node resources, provide information descriptive of the cloud node resources to a customer through the terminal, receive a reservation for cloud node resources from the customer, store the reservation in the data storage, determine assignments of the cloud node resources for the reservation and any other pending reservations according to one or more objectives, and allocate the cloud node resources to customers according to the resource assignments.

Computer cluster with objective-based resource sharing

Computer-implemented systems and associated operating methods implement a fast join for databases which is adapted for usage with flash storage. A system comprises a processor that performs a join of two tables stored in a storage in pages processed in a column orientation wherein column values for all rows on a page are co-located in mini-pages within the page. The processor reduces input/output operations of the join by accessing only join columns and mini-pages containing join results.

Database join optimized for flash storage

A method of assigning resources of a computer duster with resource sharing according to objectives. The method includes monitoring resources of each of a plurality of cloud nodes, providing information descriptive of the cloud node resources, receiving a reservation, determining whether resources are available to satisfy the reservation and any other pending reservations, if resources are available, using a rapid search to determine resource assignments for the reservation and any other pending reservations according to one or more objectives, and allocating resources according to the resource assignments.

Resource sharing in computer clusters according to objectives

A method of operating a query system includes retrieving objects from a data source, wherein each of the retrieved objects includes (i) data and (ii) metadata describing the data. The method includes dynamically creating a cumulative schema by inferring a schema from each of the retrieved objects and merging the inferred schema with the cumulative schema. The method includes storing the data of each of the retrieved objects in a storage service. The method includes receiving, from a user, a query, and responding to the query based on data stored by the storage service.

A method of performing a multi-way join of a plurality of database relations includes executing a plurality of pipelined two-way joins with the database relations. Each two-way join has two sequential phases. In the first phase, missing attributes of the input relations that are required to evaluate a joining criterion specific to said two-way join are fetched from a non-volatile memory device, and the input relations are joined according to the criterion. In the second phase, any additional missing attributes of the input relations are fetched from the non-volatile memory device as assigned by an optimization process executed prior to commencing the multi-way join.

Growing community of inventors

San Francisco, CA, United States of America

Stavros Harizopoulos