Snowflake Inc.

Oracle International Corporation

Other

Ipventure, Inc.

June Ray Limited

Kaiser Foundation Hospitals

William H Waddington

Thierry Cruanes

Jiaqi Yan

Abdul Q Munir

Jeffrey Rosen

Benoit Dageville

Prasanna Rajaperumal

Alexander Miller

Chen Luo

Patricia C Grewell

Zhaohui Zhang

Martin Hentschel

Boris Klots

Peter Ham

Jeffrey Ira Cohen

Khaled Yagoub

Wumengjian Zhu

Allison Waingold Lee

Jonathan D Klein

Cetin Ozbutun

Yongsik Yoon

Sangyong Hwang

Leng Leng Tan

Nicola Dan Onose

Li Yan

Boyung Lee

Xuelai Cui

Subramanian Muralidhar

Shrikanth Shankar

Dimitrios Tsirogiannis

Nikolaos Romanos Katsipoulakis

Lishi Jiang

Shivani Gupta

Daniel Leary

Adrian Hamza

Cristian Diaconu

Kumar Rajamani

Wei Huang

Hakan Jakobsson

Jaebock Lee

Bhaskar Ghosh

Gianfranco Putzolu

Michael Charles Depledge

Leonidas Galanis

Gabriel Kliot

Chandrasekharan Iyer

Gary Hallmark

Angela Amor

Philip Yam

David M Alpern

Allen Brumm

Mark D Callaghan

Prayag Chandran Nirmala

Guhan Viswanathan

John Leo

Gerry Perham

Mariusz Kwiczala

Marco H Kroll

Alan Choi

Susan L Griese

Jyotin Gautam

Wei Z Li

Mingze Xu

Ruji Xie

Shijie Xu

The subject technology receives a key-value pair. The subject technology determines a value size of the key-value pair is greater than a threshold size for an out of line blob (OLB) key space. The subject technology performs a write operation of a key-pointer pair into a regular space. The subject technology performs a plurality of write operations for multiple key-value parts of the key-value pair into the OLB key space. The subject technology performs conflict checking of the regular space for the key-value pair.

Optimizations to read and write transactions for large values in distributed databases

The subject technology receives a query, the query comprising a set of query statements. The subject technology determines that a set of resources to be utilized by the query during execution is provided by a slot, the slot comprising an allocation of resources provided by a virtual warehouse. The subject technology performs a first scheduling process for execution of the query using the slot. The subject technology determines that an available slot is provided by the virtual warehouse. The subject technology executes, using the virtual warehouse, the query using the available slot.

Future scheduler for database systems

The subject technology performs, using asynchronous threads of execution nodes of a distributed database, asynchronous transformation and compaction of key-value pairs of data that correspond to a plurality of transactional queries, the asynchronous threads transforming the key-value pairs from an initial format to a commit format that includes a commit time of data committed to the distributed database, the asynchronous threads compacting the key-value pairs by deleting the key-value pairs in the initial format. The subject technology processes, using transactional threads in the execution nodes, the plurality of transactional queries to generate results data, the asynchronous threads performing the asynchronous transformation and compaction while the transactional threads generate the results data. The subject technology stores the results data.

Scalable compaction for a distributed database

The subject technology serializes, by at least one hardware processor, non-primary key data of column-organized data into compressed serialized value data that is in a row-organized sequence, the compressed serialized value data compressed using at least one bitmap, the non-primary key data comprising a schema identifier, the column-organized data being stored in a columnar database system, the column-organized data comprising primary key data and the non-primary key data. The subject technology stores the compressed serialized value data in a key-value data store of a key-value database system, the key-value database system processing key-value data in a key-value format. The subject technology receives a query by the columnar database system. The subject technology deserializes a portion of the compressed serialized value data that corresponds to the query. The subject technology processes the query using the columnar database system.

Serialization of data in a concurrent transaction processing distributed database

The subject technology generates, by a compute service manager, a schema hash value for a new schema version associated with a new schema version value, the schema hash value based on determining a sum of hash values of a set of attributes of value columns, the set of attributes comprises a column identifier, and a logical type of a column. The subject technology stores a mapping of the schema hash value to the new schema version value for a table in a metadata database. The subject technology stores a new schema entry based on the schema hash value, the new schema version value, and a new column for the table in the metadata database, the metadata database storing multiple entries for different schema versions, each entry including a particular schema hash value for mapping to a corresponding schema version from the different schema versions.

Schema evolution for key columnar data into row-organized sequences

The subject technology receives a statement to initiate an online index building process of an index. The subject technology determines that the index can be generated. The subject technology performs a write operation to store an index record indicating an initial status of the online index building process. The subject technology performs a statement fencing process. The subject technology performs a back-filling process of the index. The subject technology validates the index.

Online index creation for multi-version database without table locks

The subject technology determines that a first visible version of a key is not one of a N number of newest versions of the key based on searching for the key in a regular space to locate the first visible version of the key. The subject technology performs a search of an undo space to locate a second visible version of the key. The subject technology determines whether the first visible version or the second visible version of the key is newer. The subject technology provides a newer version of the key between the first visible version and the second visible version of the key.

Long-lived statements optimizations in a database system

A method includes decoding, by at least one hardware processor, an enqueue request received from a data process of a database system. The enqueue request includes a task. The task is enqueued in an in-memory task queue. An enqueue acknowledgment is encoded for transmission to the data process responsive to the enqueue request. The task is persisted in a storage location associated with the in-memory task queue. Initiate a lease of the task to a worker node in response to a lease request received from the worker node. A dequeue request is received from the worker node where the dequeue request indicates completion of the task by the worker node. The task is dequeued from the in-memory task queue based on the dequeue request.

Asynchronous task queue configuration in a database system

The subject technology receives a query, the query including a set of statements for performing the query. The subject technology populates a compilation context based at least in part the query. The subject technology invokes a compiler to perform a compilation process based on the compilation context. The subject technology performs a lookup operation on a stored plan cache for an exact match based on information from the compilation context. The subject technology, in response to determining an exact match, determines whether the particular query plan requires re-compilation based on a data dependent optimization. The subject technology determines whether a plan cache entry corresponding to the particular query plan includes a data property constraint. The subject technology determines whether the data property constraint still holds based on a set of data properties.

Query plan cache in database systems

The subject technology receives a query, the query including a set of statements, the set of statements including a first statement comprising a first data manipulation language operation. The subject technology determines that the set of statements includes at least one statement defining a first constraint, and a second statement indicating that the first constraint is statement deferrable for performing validation of the first constraint. The subject technology generates a key value constraint check operator as a node in a query plan. The subject technology generates a key value DML operator as a second node in the query plan, the key value DML operator corresponding to the first DML operation. The subject technology performs, using the key value constraint check operator, a validation process of the first constraint based at least in part on a set of violating keys of each DML operation from the set of statements.

Deferred constraints support in distributed database systems

Resource provisioning systems and methods are described. In an embodiment, a system includes a plurality of shared storage devices collectively storing database data, an execution platform, and a compute service manager. The compute service manager is configured to determine a task to be executed in response to a trigger event and determine a query plan for executing the task, wherein the query plan comprises a plurality of discrete subtasks. The compute service manager is further configured to assign the plurality of discrete subtasks to one or more nodes of a plurality of nodes of the execution platform, determine whether execution of the task is complete, and in response to determining the execution of the task is complete, store a record in the plurality of shared storage devices indicating the task was completed.

Resource provisioning in database systems

The subject technology inserts, by a first transaction, a new version of an object, the first transaction including a first statement to perform an update operation to a row in a first table, the object corresponding to data in the row to be updated, the first statement including information comprising an object key associated with the object. The subject technology performs, by a second transaction, a range read, the range read including information indicating the object key. The subject technology receives a set of conflicting transactions from the range read. The subject technology determines that a conflict occurred between the first transaction and a third transaction from the set of conflicting transactions. The subject technology performs a restart of the first transaction in response to determining that the conflict occurred.

Concurrency control for transactions in database systems

The subject technology performs a locking operation on a first set of keys by a first statement of a first transaction. The subject technology determines that a conflict occurred between the first statement and a second transaction. The subject technology determines that the second transaction has yet to complete after a predetermined period of time. The subject technology performs a deadlock detection process where the subject technology stores a key and value in a table indicating the first transaction and the second transaction, detects, based at least in part on a graph traversal of the table starting from the first transaction, a cycle between the first transaction and the second transaction, and determines that the first transaction is a youngest transaction in the detected cycle. The subject technology ceases execution of the first transaction in response to the first transaction being a youngest transaction in a detected cycle.

Deadlock detection in distributed databases

A distributed database system can implement a column-based database system and a row-based database system for processing data. The row-based database system can store data organized into key value pairs, and data to be processed by the row-based database system is converted to a key-value format compressing keys that correspond to values. The distributed database system can perform serialization and compression in converting the data to the key-value format for efficient data storage performance. The distributed database system can unpack portions of the converted serialized compressed data in response to queries that process a portion of serialized compressed data without unpacking the entire converted dataset.

The subject technology receives a first transaction, the first transaction to be executed on linearizable storage. The subject technology assigns, prior to performing a read operation, a first read version to the first transaction, the first read version indicating a first version of the linearizable storage, the first read version corresponding to a particular timestamp of a current time. The subject technology performs the read operation from the first transaction on a table in a database. The subject technology determines a first commit version identifier corresponding to first data resulting from the read operation. The subject technology proceeds to perform a particular write operation from the first transaction. The subject technology determines that the first commit version identifier fails to match a second commit version identifier corresponding to a second transaction. The subject technology aborts the particular write operation from the first transaction.

Managing concurrent transactions in database systems

Growing community of inventors

Foster City, CA, United States of America

William H Waddington