Snowflake Inc.

Ashish Motivala

Benoit Dageville

Thierry Cruanes

Marcin Zukowski

Martin Hentschel

Yi Fang

Yizhi Zhu

Spyridon Triantafyllis

Spyros Triantafyllis

Spyros Triantafylis

A system and method for pruning data based on metadata. The method may include receiving a query that includes a plurality of predicates and identifying one or more applicable files including database data satisfying at least one of the plurality of predicates. The identifying the one or more applicable files including reading metadata stored in a metadata store that is separate from the database data. The method further includes pruning inapplicable files including database data that does not satisfy at least one of the plurality of predicates to create a reduced set of files and reading the reduced set of files to execute the query.

Data pruning based on metadata

A system and method of using stateless nodes to process data of catalog objects. The method includes accessing a catalog object including metadata associated with a dataset. The method includes determining, based on the metadata, whether a stateless node is to process the dataset. The method includes detecting, by one or more processors, a failure associated with the stateless node in response to determining that the stateless node is to process the dataset. The method includes replacing the stateless node with a different node without recreating a particular state.

Using stateless nodes to process data of catalog objects

Example systems and methods for cloning catalog objects are described. In one implementation, a method includes creating a second catalog object by mapping a second portion of second metadata of the second catalog object to same data of a same data file as a first portion of first metadata of a first catalog object, and, in response to a data storage or data retrieval request directed to the second catalog object, deleting data associated with the second portion of the second metadata from the second catalog object independently of the first catalog object.

Cloning catalog objects

Example systems and methods for cloning catalog objects are described. In one implementation, a method includes creating a copy of a catalog object without copying a data associated with the catalog object by only coping metadata associated with the object. The method further includes modifying, by one or more processors, the data associated with the catalog object independently of the copy of the catalog object.

Accessing data of catalog objects

A system, apparatus, and method for processing queries wherein the query includes a request to access or delete data and accessing metadata associated with the set of data, the metadata defining data characteristics of the set of data and identifying at least sets of data that need or not need to be accessed or deleted based on the metadata without accessing the actual data in the set of data; also methods to optimize processing of some operations based on the collected metadata on data.

The subject technology receives first metadata corresponding to a set of micro-partitions. The subject technology stores a first data structure and a second data structure in storage as a first file and a second file, first data structure including the first metadata and a second data structure including second metadata, the first metadata corresponding to a set of micro-partitions, the second metadata for a grouping of the first metadata, the second data structure including information associating the second metadata to the first metadata. The subject technology stores third metadata for a table, the third metadata comprising information about data stored in a micro-partition of the table.

Multi-level data for database systems

Example systems and methods for cloning catalog objects are described. In one implementation, a method identifies an original catalog object associated with data and creates a duplicate copy of the original catalog object without copying the data itself. The method allows access to the data using the duplicate catalog object and supports modifying the data associated with the original catalog object independently of the duplicate catalog object. The duplicate catalog object can be deleted upon completion of modifying the data associated with the original catalog object.

Example systems and methods for cloning catalog objects are described. In one implementation, a method includes copying first metadata of a first catalog object to create second metadata of a second catalog object such that a second portion of the second metadata and a first portion of the first metadata are associated with same data of a same data file, and generating a response to a data storage or data retrieval request directed to the first catalog object by accessing the data file using the second metadata of the second catalog object.

A system and method for pruning data based on metadata. The method may include receiving a query with a plurality of predicates and identifying one or more applicable files that includes database data satisfying at least one of the plurality of predicates. The identifying the one or more applicable files including reading metadata stored in a metadata store that is separate from the database data. The method further includes pruning inapplicable files comprising database data that does not satisfy at least one of the plurality of predicates to create a reduced set of files and reading the reduced set of files to execute the query.

Example systems and methods for cloning catalog objects are described. In one implementation, a method includes creating a copy of a catalog object without copying a data associated with the catalog object by only coping metadata associated with the object. The method further includes modifying, by one or more processors, the data associated with the catalog object independently of the copy of the catalog object such that modified data is not visible to the copy of the catalog object and wherein the modified data is not visible to the catalog object in a database. The method further includes deleting the copy of the catalog object following completion of the modification of the data associated with the catalog object.

A system and method for pruning data based on metadata. The method may include receiving a query that includes a plurality of predicates and identifying one or more applicable files including database data satisfying at least one of the plurality of predicates. The identifying the one or more applicable files including reading metadata stored in a metadata store that is separate from the database data. The method further includes pruning inapplicable files comprising database data that does not satisfy at least one of the plurality of predicates to create a reduced set of files and reading the reduced set of files to execute the query.

Growing community of inventors

Foster City, CA, United States of America

Ashish Motivala