Snowflake Inc.

Chong Han

Igor A Zinkovsky

Nithin Mahesh

Elliott Brossard

Srilakshmi Chintala

Nitya Kumar Sharma

Albert L Hu

Yanrui Zhang

Saurin Shah

Robert Bengt Benedikt Gernhardt

Ganeshan Ramachandran Iyer

Aravind Ramarathinam

Jiaxing Liang

Abdullah Al Mahmood

The distributed database can implement unstructured data replication using an internal or external storage location. Metadata, such as a directory table that lists the unstructured files, can be replicated across different deployments, followed by replication of the staged data. Replicating the staged data can be implemented by replication of only the stage metadata or replication of the database files between the deployments.

Replication of unstructured staged data between database deployments

Multiple execution environments are established for processing database statements with diverse package dependencies. A database system receives a user application and identifies, via hardware processors, multiple package dependencies required by the application, including at least a first and a second package dependency. Upon receiving a database statement from a user that invokes these dependencies, the system retrieves the necessary packages from a repository. It then creates a first execution environment incorporating the first package dependency and a second execution environment with the second package dependency. Utilizing the multiple execution environments, the system generates results data in response to the database statement, effectively managing and executing user-defined functions that rely on different versions or types of package dependencies within a unified database framework.

Multiple environments and functions in a database system

A method includes detecting, by at least one hardware processor, an upload of a user application within a database system. The user application includes user-defined function (UDF) code. A plurality of dependencies of the user application is determined by the at least one hardware processor. A plurality of execution environments corresponding to the plurality of dependencies is generated by the at least one hardware processor. The plurality of execution environments is associated with a corresponding plurality of data types of the UDF. A database query is decoded. The database query specifies database data of a data type of the plurality of data types of the UDF. The database query is processed using at least one of the plurality of execution environments to generate results data. The at least one of the plurality of execution environments corresponds to the data type of the plurality of data types of the UDF.

Processing user-defined functions (UDFs) using multiple execution environments

Embodiments of the present disclosure provide techniques for deduplicating files during internal stage replication using a directory table of the replicated internal stage that is modified as a cache for storing and retrieving original file-level metadata for the replicated files. An initial list of candidate files for loading from the internal stage to a table of the target deployment is prepared based on the files listed in the internal stage, and refined using a directory table lookup. If there is any inconsistency between the files registered in the directory table and the files listed in the internal stage, the target deployment will inspect the user-defined file-level metadata to obtain original file-level metadata for each file that is present in the internal stage but not in the directory table. This information may be used during deduplication to ensure that no duplicate files are loaded.

Load history calculation in internal stage replication

A database dependency resolver system can identify different dependencies of a user application and integrate the identified dependencies in different execution environments of a distributed database system. The different execution environments can manage different versions of a given programming language, or other types of computational architectures (e.g., different CPU types). A database user can provide a database statement (e.g., query) that activates the different dependencies in the different environments to generate results data.

Multiple environments and functions in a distributed database

A database query specifies a user-defined function (UDF) to apply to one or more tables in a distributed database. The one or more tables are identified in the distributed database. The one or more tables are associated with an initial set of data types native to the distributed database. The one or more tables are converted to database data comprising data types of the UDF. The UDF is executed on the database data to generate results data. The results data is converted to table results data. The table results data is associated with the initial set of data types. The table results data is stored in the distributed database.

Growing community of inventors

Bellevue, WA, United States of America

Chong Han