Microsoft Technology Licensing, LLC

Yongfu Lou

Maneesh Sah

Jegan Devaraju

Shane Kumar Mainali

Pradeep Seela

Pei Zhang

Yi-Fan Tsai

Kristopher T Lange

Arild Einar Skjolsvold

Krishnan Varadarajan

Qiuhan Wang

Donglin Wei

Data is replicated between primary and secondary storage systems using a data pulling process or a log shipping process. If data is to be replicated through the log shipping process, a transaction and its associated data get stored in a transaction log in the secondary storage system. Subsequently, when the transaction gets executed, actual data gets persisted from the transaction log to a data log in the secondary storage system. If the data is to be replicated through the data pulling process, the transaction is stored in the transaction log. However, the associated data for the transaction is retrieved from the first primary storage system and stored directly in the data log in the secondary system.

Data replication in a distributed storage system

A distributed storage system includes multiple partitions. A block received for storage is striped across the partitions if it meets predetermined criteria. Multiple tables are maintained for indexing the blocks and the subblocks of blocks that are partitioned. The tables for the subblocks and the tables for the corresponding blocks of the subblocks are updated to include metadata for the subblocks for retrieving the subblocks in response to a request for a corresponding stored block.

Data partitioning in a distributed storage system

Live migration of partitions can be achieved using server-to-server communications to coordinate and control partition migration and rebuild a persistent state of the partition on a destination server while the partition is served on the source server. The live migration approach breaks the heavy dependency on the partition master by eliminating most of the server-to-master communications, delegating coordination and control of the migration to the source and destination servers. After the migration is initiated by the partition master, the source server sends in-memory cache entries indicating the persistent state of the partition to the destination server. Once transferred, the two servers can cooperate to complete the handover, which loads the partition on the destination server using the cache entries received from the source server. Once the source server offloads the partition, it can redirect traffic for the partition to the destination server by triggering an update to the partition map.

Live migration of partitions

Prefetching metadata stream sub-portions and/or log stream sub-portions to recreate partition states in a distributed computing system. A system monitors performance metrics associated with a first node running an instance of a partition and, based thereon, cause a second node to duplicate a desired operational state of the partition. The desired operational state may result from transactions being performed on the partition after a previous operational state. Re-creating the desired operational state may include causing multiple prefetch buffers to simultaneously load sub-portions of the transactions and then commencing a replay operation after an initial subset of the transactions is loaded but prior to the entirety of the transactions being fully loaded. The system may preemptively analyze individual prefetch buffers to identify partial buffers that have not fully loaded. Then, the system may cause other prefetch buffers to assist the partial buffers in fully loading a corresponding subset of transactions.

Parallel prefetching log/meta stream sub-portions to recreate partition states in a distributed computing system

Growing community of inventors

Redmond, WA, United States of America

Yongfu Lou