Microsoft Technology Licensing, LLC

Alibaba Group Holding Limited

Alibaba Innovation Private Limited

Yikang Xu

Bradley Gene Calder

Aaron William Ogus

Rui Wang

Shane Kumar Mainali

Ju Wang

Andrew James Edwards

Lon-Chan Chu

Peng Xu

Huseyin Simitci

Leonard Alan Collins

Yu Du

Cheng Huang

Ioan Oltean

Ido Miles Ben-Shachar

Jiesheng Wu

Ivan J Leichtling

Kenneth Fern

Ryan W Battle

Qingda Lu

Claus T Giloi

Austina M De Bonte

Christer Lundin

Peter Ollodart

Kristofer Noel Iverson

Krishnan Varadarajan

John S Holmes

Jegan Devaraju

Haiyong Wang

Xin Liu

Vick Bhaskar Mukherjee

John T Spivey

Manish Chablani

Muhammad Junaid Shahid

Pradeep Seela

Liping Bao

Qinghua Ye

Zhu Pang

Shuo Chen

Jegan Devanraju

A client device may transmit a data stream including an object (such as a data file or record) to a storage system for storing the object in a storage device. In response to receiving the data stream, the storage system may store or write data of the object into a plurality of logical blocks of the storage device with an end-to-end data protection based at least in part on a comparison of a combination of check codes of a header of the object, the object and padding data with a combination of a plurality of check codes that are generated separately for metadata associated with the object and corresponding parts of the object stored in the plurality of logical blocks. The storage system may further provide an end-to-end data protection for reading data of an object stored in multiple logical blocks of a storage device.

End to end check code protection in storage engine

A key-value store is provided, implementing multiple-tiered sorted data structures in memory and storage, including concurrent write buffers in memory, and page-level consolidation of updates on storage, where pages are trivially translated in physical-to-virtual address mapping. The key-value store is built on an indexed sorted data structure on storage, occupying much less storage space and incurring much less disk activity in consolidating updates than a conventional log-structured merge tree organized into files. Concurrent write buffers operate concurrently and independently so that data is committed from memory to storage in an efficient manner, while maintaining chronological sequence of delta pages. Trivial mapping allows mappings of a number of physical pages to be omitted, enabling page mapping tables to occupy less storage space, and simplifying processing workload of read operation retrievals from storage.

High-performance key-value store

Methods and systems for in-memory metadata reduction in cloud storage system are provided. According to an aspect, a method comprises receiving a first command to write a data stream to a storage device; writing the data stream into a plurality of fragments having logical addresses corresponding to physical addresses on the storage device; and generating an index for individual fragment of the plurality of fragments, the index indicating information to locate the physical addresses of the individual fragment. Individual records in the individual fragment have a same pre-set logical size and all individual records in the individual fragment are continuous, and the index indicates the information including at least: an offset value of the individual record in the individual fragment; the pre-set logical size of the individual record; and a pre-set physical size of the individual record.

Method and system for in-memory metadata reduction in cloud storage system

A storage engine may obtain one or more object access properties of an object to be received, and determine a type of storage device that is suitable or desirable for storing the object from among different types of storage devices based at least in part on the one or more object access properties of the object to be received. In response to determining the type of storage device, the storage engine may allocate a storage device of such type for the object. The storage engine may then receive the object, and store the object into the allocated storage device.

Using object properties for facilitating object storage

When performing a recycling operation on a storage device, a storage system may use or create a data buffer in the storage device, and designate the data buffer to temporarily store data of data blocks to be recycled in the storage device using direct memory access (DMA) operations that are performed internally in the storage device, without the need of reading the data of the data blocks from the storage device and writing the data into a host memory of the storage system, thereby saving or reducing the consumptions of the communication bandwidth of a communication channel between the storage system and the storage device, and the memory bandwidth of the host memory.

Recycle optimization in storage engine

In various embodiments, methods and systems for implementing a distributed metadata management system in distributed storage systems are provided. A distributed storage system operates based on data storage resources (e.g., extents and streams). The distributed metadata management system is implemented for extent and stream metadata to facilitate the scalability of metadata processing. The distributed storage system implements extent managers and stream managers that independently manage extent and stream metadata, respectively. The extent managers are associated with an extent table that stores extent metadata. The stream managers are associated with streams that store associations with extents. The distributed metadata management system can also utilize a bootstrap layer that leverages components of a legacy distributed storage system to facilitate distributed management of extent and stream metadata. The bootstrap layer is used to store the extent table as a system table and to persist the state of the stream manager as system streams.

Distributed metadata management in a distributed storage system

Methods and systems are provided for improving a journaling file system by reducing journaling overhead utilizing a remapping interface to log-structured cloud block devices such that blocks in a journaling transaction are atomically mapped to their final disk locations with minimum overhead. With log-structured cloud block devices, all blocks may be stored in logfiles and indexed for access. By modifying the index of the destination block to point to the corresponding journal block, the additional write may be avoided, and a journal block may effectively be remapped from its original LBA to its destination LBA without any actual data movement, thereby reducing the file system journaling overhead.

Journaling overhead reduction with remapping interface

In various embodiments, methods and systems for implementing garbage collection in distributed storage systems are provided. The distributed storage system operates based on independent management of metadata of extent and stream data storage resources. A hybrid garbage collection system based on reference counting garbage collection operations and mark-and-sweep garbage collection operations is implemented. An extent lifetime table that tracks reference weights and mark sequences for extents is initialized and updated based on indications from extent managers and stream managers, respectively. Upon determining that an extent is to be handed-off from weighted reference counting garbage collection operations to mark-and-sweep garbage collection operations, a reference weight field for the extent is voided and a mark sequence field of the extent is updated. The mark sequence field is updated with a latest global sequence number. The mark-and-sweep garbage collection operations are utilized to reclaim the extent when the extent is no longer referenced.

Hybrid garbage collection in a distributed storage system

In various embodiments, methods and systems for implementing multiple transaction logs in a distributed storage system are provided. A log stream component detects performance metrics of a plurality of log streams. The performance metrics are associated with requests from partitions in the distributed storage system. A transaction component receives a request to execute a transaction using a log stream. The request is received from a partition of the distributed storage system. The performance metrics of the plurality of log streams can be referenced, where the performance metrics indicate a performance capacity of a selected log stream to process the request. A log stream for executing the transaction is determined based on the performance capacity. The log stream selected can also factor request attributes of the request. The transaction component communicates the request to be executed, using the log stream to perform the transaction.

Multiple transaction logs in a distributed storage system

Techniques are described that enable enterprise management of public instant message (IM) communications. When a user requests a connection with a public IM service, the connection request is redirected to a gateway server that is associated with a domain specified in the user's user ID. The gateway server acts as a pass through server between the user's IM client application and the public IM service. The gateway server may be configured to log IM communications in which the user participates. The gateway server may also be configured to control whether or not the user is able to participate in point-to-point communications through the user's IM client application. An enterprise routing server may also be implemented in a geo-distributed enterprise to route a redirected public IM connection request to an enterprise IM gateway server based on a geographic location associated with the user.

Enterprise management of public instant message communications

Embodiments of the present invention relate to systems, methods and computer storage media for erasure coding data in a distributed computing environment. A sealed extent is identified that is comprised of two or more data blocks and two or more index blocks. The sealed extent is optimized for erasure coding by grouping the two or more data blocks within the optimized sealed extent together and grouping the two or more index blocks within the optimized sealed extent together. The optimized extent may also be erasure coded, which includes creating data fragments and coding fragments. The data fragments and the coding fragments may also be stored in the distributed computing environment. Additional embodiments include monitoring statistical information to determine if replication, erasure coding or a hybrid storage plan should be utilized.

Erasure coding immutable data

A strategy is described for constructing bloom filter information and exception information. The bloom filter information is constructed to express a set of items in a lossy compressed form. The exception information reveals occasions in which the bloom filter information erroneously indicates that a candidate item is a member of the set. The strategy can apply the bloom filter information and the exception information to select a representative device among a group of devices on which a user may maintain simultaneous presence.

Strategies for compressing information using bloom filters

Conferencing data is reliably delivered to computer systems participating in a hierarchically arranged multi-cast conferencing session. When a child computer system does not receive a multi-cast packet (e.g., an IP multi-cast packet), the child computer system sends a negative acknowledgment to a parent computer system. In response, the parent computer system re-transmits conferencing data that was contained in the multi-cast packet to the child computer system. Conferencing data can be re-transmitted to the child computer system via uni-cast (e.g., TCP). Accordingly, conferencing data that is not received or that is damaged via multi-cast can be repaired via uni-cast. Computer systems can join an existing multi-cast conference session without having to communicate with the root computer system. The root computer system adjusts a multi-cast send rate to compensate for changed network conditions.

Reliable delivery of multi-cast conferencing data

In order to minimize the delay of the video images viewed by a network conference attendee, an intelligent buffering process (IB process) selectively discards video frames from at least one point, and in an embodiment, many network points, such as at 'in' and/or 'out' buffers of clients, servers, routers, etc. Packets of video frame data arrive at a buffer, which can fill to a predetermined limit if the packets cannot be forwarded due to delays or slow connections. To forward the most current video images, old frames in the buffer are discarded rather than forwarded. In a particular embodiment, when the buffer is full, the next arriving delta frame packet is discarded. To avoid distortion, each subsequent delta frame is discarded until a new key frame eventually arrives. If the buffer is still full when the key frame arrives, the buffer is purged and the new key frame is added.

Intelligent buffering process for network conference video

Growing community of inventors

Redmond, WA, United States of America

Yikang Xu