Google Inc.

Thibaud Baptiste Hottelier

Anoop Kochummen Johnson

Justin Levandoski

Yuri L Volobuev

Gaurav Saxena

Deepak Choudhary Nettem

Aleksandras Surna

Mingge Deng

Bigang Li

Shuang Guan

Mosha Pasumansky

Kwanho Ryu

Daniel Thurman Asheghian

Jonah Tang Soon Yuen

Victor Sergeyevich Agababov

Aspects of the disclosure are directed to merging data lake openness with scalable metadata for managed tables in a cloud database platform, allowing for atomicity, consistency, isolation, and durability (ACID) transactions, performant data manipulation language (DML), higher throughput stream ingestion, data consistency, schema evolution, time travel, clustering, fine-grained security, and/or automatic storage optimization. Table data is stored in various open-source file formats in cloud storage while physical metadata of the table data is stored in a scalable metadata storage system.

Managed tables for data lakes

A method for unstructured data analytics in data warehouses includes receiving an unstructured data query from a user, the unstructured data query requesting the data processing hardware determine one or more unstructured data files stored at a data repository that match query parameters. The method includes determining, using an object table, a set of unstructured data files stored at the data repository that matches the query parameters. The object table includes a plurality of rows, each row of the plurality of rows associated with a respective unstructured data file stored at the data repository, and a plurality of columns, each column of the plurality of columns comprising metadata associated with the respective unstructured data file of each row of the plurality of rows. The method includes returning, to the user, a structured data table including the determined set of unstructured data files.

Unstructured data analytics in traditional data warehouses

A method for optimizing a database query includes receiving a database query from a user requesting a database to conditionally return one or more rows stored at the database and characterizing one or more user filters. Each user filter filters rows from the database. The method includes determining that a security filter restricts the user from accessing one or more rows of the database. For each respective user filter, the method includes classifying the respective user filter as a safe user filter incapable of reporting an error or an unsafe user filter capable of reporting an error. The method also includes determining a filter execution order of the one or more user filters and the security filter based on the classified one or more user filters and executing, using the determined filter execution order, the one or more user filters and the security filter.

Performance optimizations for row-level security filters

Aspects of the disclosure are directed to a metadata cache for extending data warehouse features to data lakes. The metadata cache can accelerate query execution by directly accessing unmanaged data from the data lake rather than accessing the data through the data warehouse. The metadata cache can allow for filtering the unmanaged data to improve the speed of retrieving data for executing a query.

Query acceleration and metadata caching

Growing community of inventors

Mountain View, CA, United States of America

Thibaud Baptiste Hottelier