Snowflake Inc.

Robert Bengt Benedikt Gernhardt

Eric Robinson

Nithin Mahesh

Mikhail Kazhamiaka

Rodney Weaver

Sahaj Saini

Subramanian Muralidhar

Damien Carru

Martin Hentschel

Lin Chan

Tianyi Chen

Chao-Yang Lo

Vishnu Dutt Paladugu

Laxman Mamidi

Matthew Uhlar

Saurin Shah

Aravind Ramarathinam

Chong Han

Yanrui Zhang

The distributed database can implement unstructured data replication using an internal or external storage location. Metadata, such as a directory table that lists the unstructured files, can be replicated across different deployments, followed by replication of the staged data. Replicating the staged data can be implemented by replication of only the stage metadata or replication of the database files between the deployments.

Replication of unstructured staged data between database deployments

A system for data object replication includes at least one hardware processor and at least one memory storing instructions. The instructions cause the at least one hardware processor to perform operations including parsing a replication request to obtain a data object and identification information identifying a set of databases at a first deployment of a data provider. A dependency of the data object to one or more additional data objects stored in the set of databases is detected. A sequential replication of the data object and the one or more additional data object from the first deployment to a second deployment of the data provider is performed. The second deployment is identified by the replication request. A sequence of the sequential replication is based on the detected dependency.

Data object replication using database identification

A method includes generating at a source account of a database system user, a replication group object (RGO). The RGO includes a manifest identifying one or more data object types and one or more databases. A replication of data objects of the one or more data object types is performed from the one or more databases to a target account of the database system user. The replication of the data objects is performed as a unit. The RGO is replicated as a failover group object (FGO) within the target account of the database system user. A network failure is detected at the source account of the database system user. The target account of the database system user is promoted as a new source account based on the detecting of the network failure.

Failover configuration using replication group objects

This disclosure provides methods and techniques of data replication involving cross replication group (RG) references. The present disclosure avoids automatic replication failing when an entity in an RG refers to another entity external to the RG. The entity to be replicated within the RG is referred to as the 'referring entity,' and the entity as the dangling reference is referred to as the “referred entity.” Although the referring and referred entities are not replicated together in a replication operation, the referred entity may have already been replicated to the target account in another replication operation on a different replication group. In such cases, the data replication procedure may, according to aspects of the present disclosure, check if the referred entity has already been replicated, and if so, proceed to replicate the referring entity without fail, and link the referring and referred entities to enable normal functioning of the referring entity.

Data replication with cross replication group references

Different database deployments, or other data system deployments, may want to communicate with each other without sacrificing security or control. To this end, embodiments of the present disclosure may provide secure message exchange techniques for a source and/or target deployment. Configurable rule sets may be stored in the deployments; the rule sets may define what messages may be communicated between deployments. The deployments may implement a selective filtering scheme in one or more stages based on the rule sets to filter outgoing and/or incoming messages.

Secure message exchange between deployments

Techniques for executing show commands are described herein. A plurality of navigation steps is utilized, each navigation step corresponding to a different layer in a database structure and each navigation step including an operator to fetch items from a metadata database up to respective bounded limits. Dependency information is also fetched for objects of the specified object type in the show command. After a set of objects from the last layer are processed, memory for the navigation steps is flushed and the next set of objects are processed.

Catalog query framework on distributed key value store

A method includes selecting, by at least one hardware processor of a first database deployment, data for transmission to a second database deployment. The data includes a plurality of files forming a data stream. At least a first file of the plurality of files is encoded using a first encryption key which is stored in at least a second file of the plurality of files. The at least a first file is further encoded to include a second encryption key associated with at least a third file of the plurality of files. A set of nodes is identified in a virtual private network connected to the first and second database deployments. The data stream is transmitted from the first database deployment to the second database deployment via the set of nodes of the virtual private network that is connected to the first database deployment and the second database deployment.

Virtual network replication using staggered encryption

A system for data object replication includes at least one hardware processor and at least one memory storing instructions. The instructions cause the at least one hardware processor to perform operations including decoding a replication request to obtain a data object. Object dependencies associated with the data object and a plurality of additional data objects are determined. A replication sequence of the data object and the plurality of additional data objects is determined based on the object dependencies. A replication of the data object and at least one of the plurality of additional data objects is performed according to the replication sequence. A notification of a successful completion of the replication is generated.

Data object replication using a replication sequence

Provided herein are systems and methods for configuring replication group objects. A system includes at least one hardware processor coupled to memory and configured to decode a replication request received from a client device of a data provider. The replication request indicates a plurality of account objects for replication. The plurality of account objects associated with a corresponding plurality of account object types. A replication group object is generated based on the replication request. The replication group object includes a manifest. The manifest lists the plurality of account objects. The replication of the plurality of account objects is performed from a source account of the data provider into at least one target account. The replication is based on the manifest of the replication group object.

Replication group objects configuration in a network-based database system

This disclosure provides methods and techniques of data replication involving cross replication group (RG) references. The present disclosure avoids automatic replication failing when an entity in an RG refers to another entity external to the RG. The entity to be replicated within the RG is referred to as the 'referring entity,' and the entity as the dangling reference is referred to as the 'referred entity.' Although the referring and referred entities are not replicated together in a replication operation, the referred entity may have already been replicated to the target account in another replication operation on a different replication group. In such cases, the data replication procedure may, according to aspects of the present disclosure, check if the referred entity has already been replicated, and if so, proceed to replicate the referring entity without fail, and link the referring and referred entities to enable normal functioning of the referring entity.

A networked device communication system can configure network devices (e.g., a primary and secondary database) to send and receive sequences of messages, such as replicated data, using one or more keypairs and wrapping keys. The sequences of messages can include an initial set of messages that are encrypted by a wrapping key, and further include another set of messages that are encrypted by a replaced staggered key. The sequence of messages can be configured to be decrypted without exporting keys of hardware security modules.

Message-based database replication

A network device communication system can configure network devices (e.g., a primary and secondary database) to send and receive sequences messages, such as replication data, over a channel comprising a plurality of private network nodes. The messages can be generated and encrypted using one or more key pairs and changing wrapping replication keys to send and receive the messages between different types of database deployments.

Private virtual network replication of cloud databases

Provided herein are systems and methods for configuring replication of account object metadata. A system includes at least one hardware processor coupled to a memory and configured to decode a replication request received from a client device of a data provider. The replication request indicates at least a first account object, a source account, and a target account of the data provider. An object dependency of the at least first account object to at least a second account object of the data provider is determined. A replication of the at least first account object and the at least second account object is performed from the source account into the target account of the data provider.

Growing community of inventors

Seattle, WA, United States of America

Robert Bengt Benedikt Gernhardt