Microsoft Technology Licensing, LLC

Robin Dhananjay Dhamankar

Hanumantha Rao Kodavalla

Tomas Talius

Vishal Kathuria

Martin J Sleeman

Peter Byrne

Rui Wang

Alexander Evgenievich Nosov

Qun Guo

Michael Habben

Xiaowei Jiang

Satyanarayana Narlapuram

Lev Novik

Michael Nelson

Yunxin Wu

Shaoyu Zhou

Mahesh K Sreenivas

Kapil Gupta

Yixue Zhu

Kevin Farlee

Eric Ray Christensen

Michael J Purtell

Chunjia Li

Sethu M Kalavakur

Leigh M Stewart

Gayathri Venkataraman

Avraham Levy

A maximum lag between data stores can be specified that corresponds to a recovery point objective defined in a service level agreement. Lag can be monitored during a data replication between a primary data store and a secondary data store located in geographically different regions. Activity on the primary data store including incoming data transactions can be throttled as a function of the lag and the maximum lag.

Recovery point objective enforcement

Streaming database replication is provided by merging a stream of user transactions with a stream of copy transactions comprising copy data into a combined stream on a source. A target receives a single stream that includes copy transaction and concurrent user transactions in an order that enables conflicts between data being copied and user transactions to be handled correctly. Furthermore, locks applied to data subject to a copy transaction or user transaction can be released once the copy transaction or user transaction is added to the combined stream.

Data seeding optimization for database replication

Managing database recovery time. A method includes receiving user input specifying a target recovery time for a database. The method further includes determining an amount of time to read a data page of the database from persistent storage. The method further includes determining an amount of time to process a log record of the database to apply changes specified in the log record to a data page. The method further includes determining a number of dirty pages that presently would be read in recovery if a database failure occurred. The method further includes determining a number of log records that would be processed in recovery if a database failure occurred. The method further includes adjusting at least one of the number of dirty pages that presently would be read in recovery or the number of log records that would be processed in recovery to meet the specified target recovery time.

Managing database recovery time

Embodiments are directed to implementing a transitional redo phase to allow pre-redo phase data access and to determining which data pages including unprocessed transactions are to be processed during a transitional redo phase. In an embodiment, a computer system initiates an analysis phase of a database recovery that scans a database transaction log and builds a dirty page table that includes pages corresponding to unprocessed data transactions. The computer system determines from the dirty page table which pages are to be processed during the transitional redo phase, before the final redo phase. The computer system processes the determined pages during the transitional redo phase, before the final redo phase is reached and, upon completion of the system transaction undo phase, makes available those pages that were processed during the transitional redo phase, as well as those database pages that are not scheduled for processing during the final redo phase.

Tracking redo completion at a page level

Consolidating a set of tenant log streams from separate user databases into a consolidated log stream. The method includes receiving a plurality of tenant log streams from separate user data bases. The method further includes recording the plurality of tenant log streams as a consolidated log stream. The method further includes maintaining metadata about the consolidated log stream to map log records from the plurality of tenant log streams to their location in the consolidated log stream.

Log consolidation

A distributed database transaction coordination manager's outcome store may be unavailable to resolve in-doubt transactions locally, e.g., after an unclean detach, during an outcomes store restore, or during a logical database restore. To facilitate transactional consistency, AGEs and values based on them are used. Some code avoids reformatting any closed physical portion for reuse based on the segment's most recently received minimum of persisted AGEs. Some code commits any in-doubt transactions that appear on an aggregate list and aborts any not on the aggregate list. Some code writes each segment's transaction commit records to persistent storage until the segment's persisted-AGE is not less than a common-restore-AGE, and some code resolves any in-doubt transactions based on a harvested outcomes list. Outcomes may reside in a log or a persisted transactions table. A restore checkpoint may include harvested outcomes. Segments may reside on cloud nodes and/or in a shared nothing database system.

Consistent database recovery across constituent segments

Architecture that addresses an end-to-end solution for logical transactional replication from a shared-nothing clustered database management system, which uses adaptive cloning for high availability. This can be time based using a global logical timestamp. The disclosed architecture, used for refreshing stale clones, does not preserve user transaction boundaries, which is a more complex situation than where the boundaries are preserved. In such a scenario it is probable that for a given data segment no clone of the segment may contain the complete user transaction history, and hence, the history has to be pieced together from the logs of multiple different clones. This is accomplished such that log harvesting is coordinated with the clone state transitions to ensure the correctness of logical replication.

Logical replication in clustered database system with adaptive cloning

A database is queried as of any wall-clock time within a retention period, via undo that uses database snapshots and a list of page level modifications. The snapshot is user-identified, automatically generated, or extracted from a backup. The list is maintained in a transaction log by persisting page content before a page is re-used, persisting deleted rows before they are moved, persisting compensation log record undo information, and/or logging a full page. To rewind an entire database, the undo scans the transaction log in reverse LSN order and undoes all page modifications. Undo reverses reallocated pages, table truncation, and/or table deletion, as well as page-level modifications of a schema, metadata values, and/or system tables. An as-of query is handled using as-of page(s) from a sparse page file. If the sparse page file does not already contain the responsive page(s), they are created and added to it.

Database point-in-time restore and as-of query

Aspects of the subject matter described herein relate to data change ordering in multi-log based replication. In aspects, local seeds are maintained for subtransactions involved in a transaction, where each subtransaction may occur on a different node that hosts one or more database fragments involved in the transaction. When a subtransaction communicates with another subtransaction in a transaction, the subtransaction sends its local seed to the other subtransaction. The receiving subtransaction compares its local seed with the received seed and updates its local seed if the received seed is logically after its local seed. A subtransaction uses a local seed to generate sequence identifiers for changes made by the subtransaction. These identifiers allow data changes of a transaction that are made on multiple nodes to be partially ordered relative to other changes made during the transaction.

Data change ordering in multi-log based replication

To ensure that logs representative of data changes are durably written, localized storage media cache is flushed to force a write to stable storage. Log sequence number tracking is leveraged to determine if log data in a localized storage media cache is likely to have not been written to durable storage. When this condition is likely, a FLUSH_CACHE command is issued to a storage media controller to force a durable write. This allows recovery of data changes to a transactional system even when its associated storage media does not provide write-ordering guarantees. Moreover, flushing of the storage media cache can be accomplished at a frequency that provides an upper bound on a maximum time between a transaction commit and data being made durable.

Enforced transaction system recoverability on media without write-through

A system and method for serialization and/or de-serialization of file system item(s) and associated entity(ies)is provided. A file system 'item' comprises a core class which can include property(ies). Through serialization, a consistent copy of the item and associated entity(ies), such as fragment(s), link(s) with other item(s) and/or extension(s),if any, can be captured. The serialization system includes an identification component that identities entity(ies) associated with an item and a serialization component that serializes the item and associated entity(ies). The serialization component can further serialize a header that includes information associated with the item and associated entity(ies). The header can facilitate random access to the item and associated entity(ies). The serialization system can expose application program interface(s) (API's) that facilitate the copying, moving and/or transfer of an item and its associated entity(ies) from one location to another location.

Serialization of file system item(s) and associated entity(ies)

The subject invention relates to systems and/or methods that perform an authoritative and/or a non-authoritative restore of items in a data store. Backups are provided that include data associated with items and metadata related to the items that can include a history of operations and previous synchronizations to enable replications to converge after restoration. The operations that can be performed on the items include, for example, create, delete, update, move, etc. The items in the backups can be restored along with the metadata, which can be utilized to enable synchronization partners to converge based on a desired type of restoration.

Growing community of inventors

Bellevue, WA, United States of America

Robin Dhananjay Dhamankar