Google Inc.

Amazon Technologies, Inc.

Anoop Kochummen Johnson

Yuri L Volobuev

Thibaud Baptiste Hottelier

Justin Levandoski

Stefano Stefani

Anurag Windlass Gupta

Rui Liu

Ashok Kumar

Ryan McKenzie White

Anthony A Virtuoso

Rahul Sharma Pathak

Jonathan Andrew Fritz

Elie Micah Kornfield

Bigang Li

Shuang Guan

Sachin Suresh Bhat

Ajay Baliram Jadhav

Konstantin Milyutin

Balaji Kannan

Vignesh Rajamani

Nivetha Purusothaman

Parag Pramod Chaudhari

Nanheng Wu

Jalpan Randeri

Deepak Choudhary Nettem

Yishan Yang

Victor Sergeyevich Agababov

Udit Mehrotra

Sneha Bharadwaj

A method for optimizing file storage includes receiving columnar data to store at a columnar data store with columns ordered with an initial ordering. The method includes determining, based on historical access patterns for the columnar data store, an updated ordering for the columns. The method includes storing the columnar data at a first location of the columnar data store using the updated ordering. The method includes determining that the stored columnar data is to be compacted and compressing at least a portion of the columnar data using each of a plurality of compression techniques. The method includes, based on compressing the at least a portion of the columnar data, selecting one of the plurality of compression techniques. The method includes storing the columnar data at a second location of the columnar data store using the selected one of the plurality of compression techniques.

Optimizing file storage in data lake tables

Aspects of the disclosure are directed to merging data lake openness with scalable metadata for managed tables in a cloud database platform, allowing for atomicity, consistency, isolation, and durability (ACID) transactions, performant data manipulation language (DML), higher throughput stream ingestion, data consistency, schema evolution, time travel, clustering, fine-grained security, and/or automatic storage optimization. Table data is stored in various open-source file formats in cloud storage while physical metadata of the table data is stored in a scalable metadata storage system.

Managed tables for data lakes

Aspects of the disclosure are directed to a metadata cache for extending data warehouse features to data lakes. The metadata cache can accelerate query execution by directly accessing unmanaged data from the data lake rather than accessing the data through the data warehouse. The metadata cache can allow for filtering the unmanaged data to improve the speed of retrieving data for executing a query.

Query acceleration and metadata caching

Techniques for providing an on-demand development environment are described. A service of a provider network receives a request to launch a development environment, such as a notebook, from an electronic device. The service obtains an identification of a computing cluster hosted by the provider network. The service obtains an identification of a compute instance hosted within the provider network, the compute instance executing a software environment to host one or more development environments. The service causes the compute instance to launch a development environment. The service sends a message to configure the launched development environment to execute a computer program written in the development environment using the computing cluster. The service generates a token to secure communications between the electronic device and the development environment and sends the token to an originator of the request.

On-demand development environment

A distributed in-memory query processor may utilize a remote data store. Column segments may be retrieved from the remote data store and loaded into system memory of a computing node. The query processor may, in response to receiving a request to perform a query, identify columns and ranges of column values required to process the query. Partition-level schemas and cross-partition schemas may be formed for computing nodes holding column segments in memory. Computing nodes may be selected for processing the query based on a degree to which a partition or set of partitions matches data requirements of the query.

In-memory distributed database with a remote data store

Growing community of inventors

Fremont, CA, United States of America

Anoop Kochummen Johnson