Microsoft Technology Licensing, LLC

Snowflake Inc.

Other

Cristian Diaconu

Per-Ake Larson

Michael James Zwilling

Robert P Fitzgerald

Naveen Prakash

Alejandro Hernandez Saenz

Alexander Budovski

Krystyna Ewa Reisteter

Craig Steven Freedman

Rogerio Ramos

Arkadi Brjazovski

Huanhui Hu

Chen Luo

Rogério Ramos

Vaibhav Kamra

Jun Fang

Daniel Vasquez Lopez

Pradeep Madhavarapu

Raghavendra Thallam Kodandaramaih

Wumengjian Zhu

Alexandru C Chirica

Mihir Dharamshi

Joshua Slocum

Priyank Porwal

Sheetal Shrotri

Christian Damianidis

Corbin McElhanney

Hanumantha Rao Kodavalla

Rohan Kumar

Alexander Miller

Sandeep Lingam

Peter Byrne

Sarika R Iyer

Andrew McCormick

Mike Zwilling

William H Waddington

Jacob Rubin Lorch

Xin Zhang

Srinivasmurthy P Acharya

Marcel Van Der Holst

Siddharth Deepak Mehta

Ryan L Stonecipher

Zhaohui Zhang

Karthick Krishnamoorthy

Dimitrios Tsirogiannis

Mahesh K Sreenivas

Adrian Birka

Nikolaos Romanos Katsipoulakis

Vasileios Papadimos

Vikram Wakade

Srini Acharya

Michael J Purtell

Spyridon Blanas

Devin Leo Rider

Per-Åke Larson

Oleksandr Gololobov

Fayssal Martani

Robert Patrick Fitzgerald

The subject technology receives a request to perform a table scan operation of a table. The subject technology determines that the table is being accessed for an initial time. The subject technology populates a columnar cache with data of the table provided by the table scan operation. The subject technology determines a set of schema versions of a set of rows from the data of the table. The subject technology determines schema information of each schema from the set of schema versions. The subject technology generates a result rowset and a second rowset comprising a union of columns that have appeared at least once in each row. The subject technology performs deserialization of rows from the result rowset and the second rowset. The subject technology provides the rows from the result rowset and the second rowset to write to a file in a particular format.

Schema evolution support in hybrid transactional/analytical processing (HTAP) workloads

The subject technology receives, by an execution node, blob metadata from a key-value store, the blob metadata including information related to a set of blob files. The subject technology determines, by the execution node using the blob metadata, whether a copy of each of the set of blob files is stored in a local cache of the execution node. The subject technology transforms at least one blob file, retrieved from a blob store, to a second file in a column file format, the at least one blob file being in a first format that is different than the column file format, the transforming comprising at least converting a particular snapshot file from the at least one blob file to a particular set of rowsets and writing the set of rowsets into the second file in the column file format. The subject technology stores the second file in the local cache.

Columnar cache in hybrid transactional/analytical processing (HTAP) workloads

The subject technology initializes a statement for execution. The subject technology determines that the statement has been executing for longer than a minimum statement timeout. The subject technology periodically updates a read timestamp table with a new update timestamp for an entry corresponding to the statement. The subject technology determines whether the entry corresponding to the statement has been removed from the read timestamp table. The subject technology, in response to determining that the entry has not been removed from the read timestamp table, removes the entry from the read timestamp table. The subject technology provides a set of results from completing execution of the statement.

Automated tracking of oldest running statement in distributed MVCC databases

Data base performance is improved using write-behind optimization of covering cache. Non-volatile memory data cache includes a full copy of stored data file(s). Data cache and storage writes, checkpoints, and recovery may be decoupled (e.g., with separate writes, checkpoints and recoveries). A covering data cache supports improved performance by supporting database operation during storage delays or outages and/or by supporting reduced I/O operations using aggregate writes of contiguous data pages (e.g., clean and dirty pages) to stored data file(s). Aggregate writes reduce data file fragmentation and reduce the cost of snapshots. Performing write-behind operations in a background process with optimistic concurrency control may support improved database performance, for example, by not interfering with write operations to data cache. Data cache may store (e.g., in metadata) data cache checkpoint information and storage checkpoint information. A stored data file may store storage checkpoint information (e.g., in a file header).

Write-behind optimization of covering cache

The subject technology receives a query, the query including a query range for processing the query and a set of requested columns. The subject technology based on the query range, determining a set of blob files and a set of delete vectors. The subject technology for each blob file, storing each row, including the set of request columns, into an array of rowsets. The subject technology for each rowset, generating a delete bitset to at least indicate whether each row has been deleted. The subject technology for each delta file, indicate a previous row of a visible row of the delta file as being deleted based on a delete pointer of the visible row. The subject technology providing a set of rowsets, including a corresponding selection column set, as a result of the query.

Scan-based merge for analytical query processing in HTAP systems using delete vectors

Distributed database systems including compute nodes and page servers are described herein that enable separating logical and physical storage of database files in a distributed database system. A distributed database system includes a page server and a compute node, and is configured to store a logical database file that includes data and is associated with a file identifier. Each page server is configurable to store slices (i.e., subportions) of the logical database file. The compute node is coupled to the plurality of page servers and configured to store the logical database file responsive to a received command. In an aspect, such storage may comprise slicing the data comprising the logical database file into a set of slices with each being associated with a respective page server, maintaining an endpoint mapping for each slice of the first set of slices, and transmitting each slice to the associated for storage thereby.

Separation of logical and physical storage in a distributed database system

The subject technology receives a statement to initiate an online index building process of an index. The subject technology determines that the index can be generated. The subject technology performs a write operation to store an index record indicating an initial status of the online index building process. The subject technology performs a statement fencing process. The subject technology performs a back-filling process of the index. The subject technology validates the index.

Online index creation for multi-version database without table locks

The subject technology receives a query, the query including a query range for processing the query. The subject technology sends a request to a key-value store for blob metadata and a set of recent writes for the query range. The subject technology receives the blob metadata, the blob metadata including information related to a set of blob files. The subject technology determines whether the set of blob files is stored in a local cache. The subject technology, in response to at least one blob file being missing from the set of blob files, sends a request to a blob store to retrieve the at least one blob file of the set of blob files. The subject technology transforms the retrieved at least one blob file to a column file format. The subject technology stores the transformed at least one blob file in the local cache.

Distributed database systems including compute nodes and page servers are described herein that enable separating logical and physical storage of database files in a distributed database system. A distributed database system includes a page server and a compute node and is configured to store a logical database file that includes data and is associated with a file identifier. Each page server is configurable to store slices (i.e., subportions) of the logical database file. The compute node is coupled to the plurality of page servers and configured to store the logical database file responsive to a received command. In an aspect, such storage may comprise slicing the data comprising the logical database file into a set of slices with each being associated with a respective page server, maintaining an endpoint mapping for each slice of the first set of slices, and transmitting each slice to the associated for storage thereby.

Embodiments described herein detect data corruption in a distributed data set system. For example, a system comprises node(s) for processing queries with respect to a distributed data set comprising a plurality of storage segments. A write transaction resulting from a query with respect to a particular storage segment is logged in a log record that describes a modification to the storage segment. A log service provides the log record to a data server managing a portion of the distributed data set in which the storage segment is included, which performs the write transaction with respect to the storage segment. For redundancy purposes, the data server has replica(s) that manage respective replicas of the portion of the distributed data set managed thereby. For backup purposes, snapshots of the replica(s) are periodically generated. To determine a data corruption, a snapshot of one replica is cross-validated with a snapshot of another replica.

Snapshot-based data corruption detection

Brokering log records so as to prevent log records that are not yet persisted in a persistent log from being disseminated. The log records may be generated as a primary compute system performs operations. Upon receiving a request for a log record, the broker component determines whether the requested log record has been persisted in a persistent log. If the broker component determines that the log record has been persisted in the persistent log, the broker component responds to the request by causing the requested log record to be provided to the requesting entity (e.g., a secondary compute system). On the other hand, if the log record cannot yet determine that the log record has been persisted in the persistent log, the broker component prevents the log record from being provided to the requesting entity. This prevents data from being inconsistent during recovery.

Brokering persisted and unpersisted log records

Adaptive adjusting of the growth of a persistent log. The persistent log has a log record generator that adds log records to the persistent log. In addition, there are multiple log consumers that consume records from the persistent log. The log consumers publish log processing parameters with respect to the persistent log. The log processing parameters are then used to determine an appropriate adjustment in the growth of the log, which adjustments may then be executed. As an example, the log processing parameter may be a log consumption progress, in which case the log generator may be caused to slow down the generation of log records, thereby slowing the growth of the log.

Adjusting growth of persistent log

Caching storage segments (e.g., pages) loaded from a remote storage such that, during recovery, the cached loaded storage segments may be at least partially recovered without reloading the storage segments from the remote storage. During normal operation of a computing system, storage segments are loaded from remote storage into local memory of a computing system. At some point, either due to eviction of the storage segment due to aging out of the storage segment, or due to writing of the storage segment, it is determined to write at least some of the loaded storage segments into local persistent storage. In conjunction with this, the corresponding storage segment is written to a respective storage address of the local persistent storage. Also, a correlation between an identifier of the storage segment and the respective address in the persistent storage is recorded in a persistent data structure.

Persistent storage segment caching for data recovery

Growing community of inventors

Kirkland, WA, United States of America

Cristian Diaconu