Google Inc.

Amazon Technologies, Inc.

Anoop Kochummen Johnson

Justin Levandoski

Yuri L Volobuev

Thibaud Baptiste Hottelier

Gaurav Saxena

Deepak Choudhary Nettem

Stefano Stefani

Anurag Windlass Gupta

Rui Liu

Ashok Kumar

Ryan McKenzie White

Rahul Sharma Pathak

Anthony A Virtuoso

Jonathan Andrew Fritz

Mingge Deng

Elie Micah Kornfield

Bigang Li

Sachin Suresh Bhat

Shuang Guan

Sneha Bharadwaj

Ajay Baliram Jadhav

Konstantin Milyutin

Vignesh Rajamani

Udit Mehrotra

Yishan Yang

Nanheng Wu

Jalpan Randeri

Parag Pramod Chaudhari

Nivetha Purusothaman

Balaji Kannan

Victor Sergeyevich Agababov

Aspects of the disclosure are directed to merging data lake openness with scalable metadata for managed tables in a cloud database platform, allowing for atomicity, consistency, isolation, and durability (ACID) transactions, performant data manipulation language (DML), higher throughput stream ingestion, data consistency, schema evolution, time travel, clustering, fine-grained security, and/or automatic storage optimization. Table data is stored in various open-source file formats in cloud storage while physical metadata of the table data is stored in a scalable metadata storage system.

Managed tables for data lakes

A method for unstructured data analytics in data warehouses includes receiving an unstructured data query from a user, the unstructured data query requesting the data processing hardware determine one or more unstructured data files stored at a data repository that match query parameters. The method includes determining, using an object table, a set of unstructured data files stored at the data repository that matches the query parameters. The object table includes a plurality of rows, each row of the plurality of rows associated with a respective unstructured data file stored at the data repository, and a plurality of columns, each column of the plurality of columns comprising metadata associated with the respective unstructured data file of each row of the plurality of rows. The method includes returning, to the user, a structured data table including the determined set of unstructured data files.

Unstructured data analytics in traditional data warehouses

A method for optimizing file storage includes receiving columnar data to store at a columnar data store with columns ordered with an initial ordering. The method includes determining, based on historical access patterns for the columnar data store, an updated ordering for the columns. The method includes storing the columnar data at a first location of the columnar data store using the updated ordering. The method includes determining that the stored columnar data is to be compacted and compressing at least a portion of the columnar data using each of a plurality of compression techniques. The method includes, based on compressing the at least a portion of the columnar data, selecting one of the plurality of compression techniques. The method includes storing the columnar data at a second location of the columnar data store using the selected one of the plurality of compression techniques.

Optimizing file storage in data lake tables

Aspects of the disclosure are directed to a metadata cache for extending data warehouse features to data lakes. The metadata cache can accelerate query execution by directly accessing unmanaged data from the data lake rather than accessing the data through the data warehouse. The metadata cache can allow for filtering the unmanaged data to improve the speed of retrieving data for executing a query.

Query acceleration and metadata caching

Techniques for providing an on-demand development environment are described. A service of a provider network receives a request to launch a development environment, such as a notebook, from an electronic device. The service obtains an identification of a computing cluster hosted by the provider network. The service obtains an identification of a compute instance hosted within the provider network, the compute instance executing a software environment to host one or more development environments. The service causes the compute instance to launch a development environment. The service sends a message to configure the launched development environment to execute a computer program written in the development environment using the computing cluster. The service generates a token to secure communications between the electronic device and the development environment and sends the token to an originator of the request.

On-demand development environment

A distributed in-memory query processor may utilize a remote data store. Column segments may be retrieved from the remote data store and loaded into system memory of a computing node. The query processor may, in response to receiving a request to perform a query, identify columns and ranges of column values required to process the query. Partition-level schemas and cross-partition schemas may be formed for computing nodes holding column segments in memory. Computing nodes may be selected for processing the query based on a degree to which a partition or set of partitions matches data requirements of the query.

In-memory distributed database with a remote data store

Growing community of inventors

Fremont, CA, United States of America

Anoop Kochummen Johnson