Google Inc.

Thibaud Baptiste Hottelier

Yuri L Volobuev

Anoop Kochummen Johnson

Justin Levandoski

Aleksandras Surna

Bigang Li

Shuang Guan

Mosha Pasumansky

Daniel Thurman Asheghian

Jonah Tang Soon Yuen

Deepak Choudhary Nettem

Victor Sergeyevich Agababov

Kwanho Ryu

Aspects of the disclosure are directed to merging data lake openness with scalable metadata for managed tables in a cloud database platform, allowing for atomicity, consistency, isolation, and durability (ACID) transactions, performant data manipulation language (DML), higher throughput stream ingestion, data consistency, schema evolution, time travel, clustering, fine-grained security, and/or automatic storage optimization. Table data is stored in various open-source file formats in cloud storage while physical metadata of the table data is stored in a scalable metadata storage system.

Managed tables for data lakes

A method for optimizing a database query includes receiving a database query from a user requesting a database to conditionally return one or more rows stored at the database and characterizing one or more user filters. Each user filter filters rows from the database. The method includes determining that a security filter restricts the user from accessing one or more rows of the database. For each respective user filter, the method includes classifying the respective user filter as a safe user filter incapable of reporting an error or an unsafe user filter capable of reporting an error. The method also includes determining a filter execution order of the one or more user filters and the security filter based on the classified one or more user filters and executing, using the determined filter execution order, the one or more user filters and the security filter.

Performance optimizations for row-level security filters

Aspects of the disclosure are directed to a metadata cache for extending data warehouse features to data lakes. The metadata cache can accelerate query execution by directly accessing unmanaged data from the data lake rather than accessing the data through the data warehouse. The metadata cache can allow for filtering the unmanaged data to improve the speed of retrieving data for executing a query.

Query acceleration and metadata caching

Growing community of inventors

Mountain View, CA, United States of America

Thibaud Baptiste Hottelier