Microsoft Technology Licensing, LLC

Alibaba Group Holding Limited

Alibaba Singapore Holding Private Limited

Other

Alibaba Innovation Private Limited

Jiesheng Wu

Bradley Gene Calder

Shane Kumar Mainali

Shashwat Srivastav

Niranjan Nilakantan

Padmanabha Chakravarthy Uddaraju

Abdul Rafay Abbasi

Qingda Lu

Gang Cheng

Ju Wang

Arild Einar Skjolsvold

Hemal Khatri

Zhu Pang

Rui Wang

Shuo Chen

Harshawardhan Gadgil

Andrew James Edwards

Shuvabrata Ganguly

Matthew Douglas Hendel

David Allen Goebel

Yikang Xu

Maxim Mazeev

Ishai Ben Aroya

Ozan Demir

Leonidas Rigas

Jaidev Haridas

Jean Ghanem

Qinghua Ye

Liping Bao

Windsor Hsu

Lei Wang

Feng Zhang

Hong Tang

Jun Song

Shunmin Zhu

Li Xu

Yuanjiang Ni

Zhixing Ren

Zhiyong Lin

Xiaohui Pei

Leilei Tian

Junpu Chen

Rong Wen

Jaiden Haridas

A key-value store is provided, implementing multiple-tiered sorted data structures in memory and storage, including concurrent write buffers in memory, and page-level consolidation of updates on storage, where pages are trivially translated in physical-to-virtual address mapping. The key-value store is built on an indexed sorted data structure on storage, occupying much less storage space and incurring much less disk activity in consolidating updates than a conventional log-structured merge tree organized into files. Concurrent write buffers operate concurrently and independently so that data is committed from memory to storage in an efficient manner, while maintaining chronological sequence of delta pages. Trivial mapping allows mappings of a number of physical pages to be omitted, enabling page mapping tables to occupy less storage space, and simplifying processing workload of read operation retrievals from storage.

High-performance key-value store

Embodiments of the present invention relate to synchronously replicating data in a distributed computing environment. To achieve synchronous replication both an eventual consistency approach and a strong consistency approach are contemplated. Received data may be written to a log of a primary data store for eventual committal. The data may then be annotated with a record, such as a unique identifier, which facilitates the replay of the data at a secondary data store. Upon receiving an acknowledgment that the secondary data store has written the data to a log, the primary data store may commit the data and communicate an acknowledgment of success back to the client. In a strong consistency approach, the primary data store may wait to send an acknowledgement of success to the client until it receives an acknowledgment that the secondary has not only written, but also committed, the data.

Synchronous replication in a distributed storage environment

Methods and systems are provided for improving user authentication and access control by a network file system service in a multi-tenant public cloud environment by receiving a request for a connection to a file system from a file system client (client), sending an identification request for identification authentication of the client to a control system, receiving a response from the control system, establishing the connection to the file system upon determining that the connection to the file system is allowed based on cloud tenant information associated with the client, receiving an attempt to access the file system from the client by a sub-user, authenticating the sub-user based on the cloud tenant information, issuing a security token including a globally unique sub-user identifier of the sub-user, and using the security token to determine access rights of the sub-user to the file system for a subsequent request.

Authentication and authorization for cloud file system

Methods and systems implement local file systems of computing nodes which translate random-access read and random-access write operation calls to append-based operation calls in accordance with a distributed file system ('DFS') implemented across storage nodes of a cloud network. Computer-executable applications running on the computing nodes may generate kernel-level read and write system calls by application programming interfaces (“APIs”) such as the Portable Operating System Interface (“POSIX”) standard, to a local file system. The local file system may translate these read and write system calls to file operations at a DFS implementing an append-only file system, as well as perform storage reclamation upon the DFS periodically and/or upon storage thresholds being exceeded. Storage for computing workloads on the computing nodes may be decoupled from individual computing nodes, in comparison to conventional architectures for computing clusters and storage clusters in cloud networks, enabling storage to be scaled without static allocations.

Distributed file system servicing random-access operations

A key-value engine may perform garbage collection for a tree or hierarchical data structure on an append-only storage device with page mappings. The key-value engine may separate hot and cold data to reduce write amplification, track extent usage using a restricted or limited amount of memory, efficiently answer queries of valid extent usage, and adaptively or selectively defragment pages in snapshots in rounds of garbage collection.

Garbage collection of tree structure with page mappings

A key-value engine of a storage system may perform a restart recovery after a system failure. The key-value engine may read a metadata log to locate a latest system checkpoint, and load a page mapping table from the latest system checkpoint. The key-value engine may replay to apply changes to the page mapping table from a system transaction log starting from a system transaction replay starting point. The key-value engine may further form one or more read-only replicas using an underlying file stream opened in a read-only mode during the recovery after the system failure to further facilitate fast recovery and provide fast response to user transactions that conduct read only transactions after the system failure.

Fast recovery and replication of key-value stores

Systems and methods discussed herein, based on a key-value data store including multiple-tiered sorted data structures in memory and storage, implement granularly timestamped concurrency control. The multiple-tiering of the key-value data store enables resolving the snapshot queries by returning data record(s) according to granularly timestamped snapshot lookup instead of singularly indexed snapshot lookup. Queries return a merged collection of records including updates from data structures in memory and in storage, such that a persistent storage transaction may refer to non-committed updates up to a timeframe defined by the snapshot read timestamp. This way, inconsistency is avoided that would result from merely reading data records committed in storage, without regard as to pending, non-committed updates thereto. The global timestamp further modifies the generation of the local transaction commit timestamp and the local snapshot read timestamp, so as to establish a granularly timestamped concurrency control scheme (over three levels of granularity).

Granularly timestamped concurrency control for key-value store

In various embodiments, methods and systems for providing access to file systems are provided. An SFA-based operation having an operation modifier for a file in a file system is received. The SFA-based operation is defined based on a file-representational state transfer (REST) interface. The SFA-REST interface comprises integrated functionality of an SFA protocol and REST protocol. The SFA-based operation is executed using the operation modifier. A REST-based operation for the file in the distributed file storage is received. The REST-based operation is defined based on the SFA-REST interface. The operation modifier is referenced in order to execute the REST-based operation. The REST-based operation based on the SFA-REST interface. A schema can be implemented for accessing the file system. The schema includes tables to store files, the table comprises fields corresponding to elements of a SFA-REST interface. A snapshot component can be implemented to backup file shares in the file system.

File service using a shared file access-rest interface

A power management system may provide power management recommendations to a computer system including a plurality of computing nodes (which may include processors, etc.), to cause the computing nodes to individually or collectively adjust power states or modes of respective processors to achieve power management of the computer system. The power management system may be provided with a power management framework that continuously utilizes direct and indirect service-level feedbacks to guide power management decisions. The power management system may employ a reinforcement learning algorithm to make power management decisions at a user level, and provide a fast decision overriding mechanism for platform events or service-requested performance boosts.

Service-level feedback-driven power management framework

Presented herein are methods of replicating versioned and hierarchical data structures, as well as data structures representing complex transactions. Due to interdependencies between data entities and a lack of guaranteed message ordering, simple replication methods employed for simple data types cannot be used. Operations on data structures exhibit dependencies between the messages making up the operations. This strategy can be extended to various types of complex transactions by considering certain messages to depend on other messages or on the existence of other entries at the data store. Regardless of origin, these dependencies can be enforced by suspending the processing of messages with unsatisfied dependencies until all of its dependencies have been met. Alternately, transactions can be committed immediately, creating entities that include versioned identifiers for each of their dependencies. These entities can then be garbage collected of the parent objects are not subsequently created.

Versioned and hierarchical data structures and distributed transactions

A mapping correspondence between memory addresses and request counts and a cache line flusher are provided, enabling selective cache flushing for persistent memory in a computing system to optimize write performance thereof. Random writes from cache memory to persistent memory are prevented from magnifying inherent phenomena of write amplification, enabling computing systems to implement persistent memory as random-access memory, at least in part. Conventional cache replacement policies may remain implemented in a computing system, but may be effectively overridden by operations of a cache line flusher according to example embodiments of the present disclosure preventing conventional cache replacement policies from being triggered. Implementations of the present disclosure may avoid becoming part of the critical path of a set of computer-executable instructions being executed by a client of cache memory, minimizing additional computation overhead in the critical path.

Random-access performance for persistent memory

Methods and systems are provided for improving a journaling file system by reducing journaling overhead utilizing a remapping interface to log-structured cloud block devices such that blocks in a journaling transaction are atomically mapped to their final disk locations with minimum overhead. With log-structured cloud block devices, all blocks may be stored in logfiles and indexed for access. By modifying the index of the destination block to point to the corresponding journal block, the additional write may be avoided, and a journal block may effectively be remapped from its original LBA to its destination LBA without any actual data movement, thereby reducing the file system journaling overhead.

Journaling overhead reduction with remapping interface

Systems and methods provided herein are directed to allowing a client to access a storage cluster, and the storage cluster to access the client, without having to communicate through the load balancer after an initial communication, or an initial packet from the client to the storage cluster, for packets belonging to the same session as the initial packet based on information included in headers of the packet.

Direct communication between physical server and storage service

Systems and methods provided herein are directed to providing both physical machine servers and virtual machines for public cloud virtual private cloud (VPC) services, which allow a user of the public cloud VPC services to selectively run an application on a physical machine server instead of a virtual machine.

Virtual private cloud services with physical machine servers and virtual machines

Growing community of inventors

Redmond, WA, United States of America

Jiesheng Wu