Snowflake Inc.

Chen Luo

Alexander Miller

William H Waddington

Zhaohui Zhang

Cristian Diaconu

Wumengjian Zhu

Mihir Dharamshi

Joshua Slocum

Corbin McElhanney

Sarvesh Singh

Fabio Angius

Andrew McCormick

Mingze Xu

The subject technology receives a key-value pair. The subject technology determines a value size of the key-value pair is greater than a threshold size for an out of line blob (OLB) key space. The subject technology performs a write operation of a key-pointer pair into a regular space. The subject technology performs a plurality of write operations for multiple key-value parts of the key-value pair into the OLB key space. The subject technology performs conflict checking of the regular space for the key-value pair.

Optimizations to read and write transactions for large values in distributed databases

The subject technology receives a request to perform a table scan operation of a table. The subject technology determines that the table is being accessed for an initial time. The subject technology populates a columnar cache with data of the table provided by the table scan operation. The subject technology determines a set of schema versions of a set of rows from the data of the table. The subject technology determines schema information of each schema from the set of schema versions. The subject technology generates a result rowset and a second rowset comprising a union of columns that have appeared at least once in each row. The subject technology performs deserialization of rows from the result rowset and the second rowset. The subject technology provides the rows from the result rowset and the second rowset to write to a file in a particular format.

Schema evolution support in hybrid transactional/analytical processing (HTAP) workloads

The subject technology receives, by an execution node, blob metadata from a key-value store, the blob metadata including information related to a set of blob files. The subject technology determines, by the execution node using the blob metadata, whether a copy of each of the set of blob files is stored in a local cache of the execution node. The subject technology transforms at least one blob file, retrieved from a blob store, to a second file in a column file format, the at least one blob file being in a first format that is different than the column file format, the transforming comprising at least converting a particular snapshot file from the at least one blob file to a particular set of rowsets and writing the set of rowsets into the second file in the column file format. The subject technology stores the second file in the local cache.

Columnar cache in hybrid transactional/analytical processing (HTAP) workloads

The subject technology initializes a statement for execution. The subject technology determines that the statement has been executing for longer than a minimum statement timeout. The subject technology periodically updates a read timestamp table with a new update timestamp for an entry corresponding to the statement. The subject technology determines whether the entry corresponding to the statement has been removed from the read timestamp table. The subject technology, in response to determining that the entry has not been removed from the read timestamp table, removes the entry from the read timestamp table. The subject technology provides a set of results from completing execution of the statement.

Automated tracking of oldest running statement in distributed MVCC databases

Techniques for multi database query processing are described. Objects located in a plurality of databases referenced in a query can be compiled. A connection string based on the compiled objects can be generated. The connection string can include mapping information related to the related to the plurality of databases and cluster information of where the plurality of databases are stored in the network-based data system. The connection string can then be included in a query plan to allow for execution of the query plan using the connection string to access the objects in the plurality of databases.

Multi database queries

The subject technology performs, using asynchronous threads of execution nodes of a distributed database, asynchronous transformation and compaction of key-value pairs of data that correspond to a plurality of transactional queries, the asynchronous threads transforming the key-value pairs from an initial format to a commit format that includes a commit time of data committed to the distributed database, the asynchronous threads compacting the key-value pairs by deleting the key-value pairs in the initial format. The subject technology processes, using transactional threads in the execution nodes, the plurality of transactional queries to generate results data, the asynchronous threads performing the asynchronous transformation and compaction while the transactional threads generate the results data. The subject technology stores the results data.

Scalable compaction for a distributed database

The subject technology receives a query, the query including a query range for processing the query and a set of requested columns. The subject technology based on the query range, determining a set of blob files and a set of delete vectors. The subject technology for each blob file, storing each row, including the set of request columns, into an array of rowsets. The subject technology for each rowset, generating a delete bitset to at least indicate whether each row has been deleted. The subject technology for each delta file, indicate a previous row of a visible row of the delta file as being deleted based on a delete pointer of the visible row. The subject technology providing a set of rowsets, including a corresponding selection column set, as a result of the query.

Scan-based merge for analytical query processing in HTAP systems using delete vectors

The subject technology receives a query, the query including a query range for processing the query. The subject technology sends a request to a key-value store for blob metadata and a set of recent writes for the query range. The subject technology receives the blob metadata, the blob metadata including information related to a set of blob files. The subject technology determines whether the set of blob files is stored in a local cache. The subject technology, in response to at least one blob file being missing from the set of blob files, sends a request to a blob store to retrieve the at least one blob file of the set of blob files. The subject technology transforms the retrieved at least one blob file to a column file format. The subject technology stores the transformed at least one blob file in the local cache.

The subject technology determines that a first visible version of a key is not one of a N number of newest versions of the key based on searching for the key in a regular space to locate the first visible version of the key. The subject technology performs a search of an undo space to locate a second visible version of the key. The subject technology determines whether the first visible version or the second visible version of the key is newer. The subject technology provides a newer version of the key between the first visible version and the second visible version of the key.

Long-lived statements optimizations in a database system

The subject technology inserts, by a first transaction, a new version of an object, the first transaction including a first statement to perform an update operation to a row in a first table, the object corresponding to data in the row to be updated, the first statement including information comprising an object key associated with the object. The subject technology performs, by a second transaction, a range read, the range read including information indicating the object key. The subject technology receives a set of conflicting transactions from the range read. The subject technology determines that a conflict occurred between the first transaction and a third transaction from the set of conflicting transactions. The subject technology performs a restart of the first transaction in response to determining that the conflict occurred.

Concurrency control for transactions in database systems

The subject technology performs a locking operation on a first set of keys by a first statement of a first transaction. The subject technology determines that a conflict occurred between the first statement and a second transaction. The subject technology determines that the second transaction has yet to complete after a predetermined period of time. The subject technology performs a deadlock detection process where the subject technology stores a key and value in a table indicating the first transaction and the second transaction, detects, based at least in part on a graph traversal of the table starting from the first transaction, a cycle between the first transaction and the second transaction, and determines that the first transaction is a youngest transaction in the detected cycle. The subject technology ceases execution of the first transaction in response to the first transaction being a youngest transaction in a detected cycle.

Deadlock detection in distributed databases

A distributed database compaction system can perform compaction for transactional queries in an asynchronous matter without affecting completion of the queries. The compaction system can implement asynchronous transformation of key pairs in the database, and older keys can be periodically purged using a scheduled compactor. Subsequent queries use the compacted stored keys to perform efficient queries with direct reads of committed transactions and more efficient access to key values stores of the distributed database.

Scalable compaction in a concurrent transaction processing distributed database

The subject technology performs a search for a key in a regular space to locate a first visible version of the key. The subject technology determines that the first visible version of the key is not one of a N number of newest versions of the key. The subject technology performs a search of an undo space to locate a second visible version of the key. The subject technology determines whether the first visible version or the second visible version of the key is newer. The subject technology provides a newer version of the key between the first visible version and the second visible version of the key.

Optimizations for long-lived statements in a database system

The subject technology inserts, by a first transaction, a new version of an object, the inserting corresponding to a first statement from the first transaction, the first statement including information indicating a first key, a first transaction identifier, a first statement number, and a first restart count. The subject technology performs, by a second transaction, a range read, the range read including information indicating the first key. The subject technology receives, after performing the range read, a set of conflicting transactions. The subject technology determines that a conflict occurred between the first transaction and a third transaction from the set of conflicting transactions.

Growing community of inventors

San Mateo, CA, United States of America

Chen Luo