Snowflake Inc.

Yanrui Zhang

Nithin Mahesh

Chong Han

Saurin Shah

Robert Bengt Benedikt Gernhardt

Ganeshan Ramachandran Iyer

Aravind Ramarathinam

Jiaxing Liang

Abdullah Al Mahmood

The distributed database can implement unstructured data replication using an internal or external storage location. Metadata, such as a directory table that lists the unstructured files, can be replicated across different deployments, followed by replication of the staged data. Replicating the staged data can be implemented by replication of only the stage metadata or replication of the database files between the deployments.

Replication of unstructured staged data between database deployments

Embodiments of the present disclosure provide techniques for deduplicating files during internal stage replication using a directory table of the replicated internal stage that is modified as a cache for storing and retrieving original file-level metadata for the replicated files. An initial list of candidate files for loading from the internal stage to a table of the target deployment is prepared based on the files listed in the internal stage, and refined using a directory table lookup. If there is any inconsistency between the files registered in the directory table and the files listed in the internal stage, the target deployment will inspect the user-defined file-level metadata to obtain original file-level metadata for each file that is present in the internal stage but not in the directory table. This information may be used during deduplication to ensure that no duplicate files are loaded.

Load history calculation in internal stage replication

Growing community of inventors

Mercer Island, WA, United States of America

Yanrui Zhang