Microsoft Technology Licensing, LLC

Shiv Kumar Rajpal

Karan Mehra

Sarosh Cyrus Havewala

Surendra Verma

Neal Robert Christiansen

Andrew M Herron

Sompong Paul Olarig

Vladimir Sadovsky

Andrea D'Amato

Sachin C Patel

Darren Glen Moss

Rajeev Y Nagar

Vyacheslav Kuznetsov

Daniel Chan

Matt Garson

Tarun Ramani

Ankur Kasturiya

Ashley P Ventura

Robin Andrew Alexander

Nathan Steven Obr

Alan M Warwick

Emanuel Paleologu

Juan-Lee Pang

Bryan Stephen Matthew

Shi Cong

Chris Lionetti

Nandagopal Kirubanandan

Gregory Robert Domzalski

Scalable data storage techniques are described. In one or more implementations, data is obtained by one or more computing devices that describes fault domains in a storage hierarchy and available storage resources in a data storage pool. Operational characteristics are ascertained, by the one or more computing devices, of devices associated with the available storage resources within one or more levels of the storage hierarchy. Distribution of metadata is assigned by the one or more computing devices to one or more particular data storage devices within the data storage pool based on the described fault domains and the ascertained operational characteristics of devices within one or more levels of the storage hierarchy.

Scalable data storage pools

A system for storing data in a dynamic fashion. The system includes a storage entity. The storage entity includes portions of a plurality of different persistent storage devices. Each storage device has a set of constraints. The storage entity is configured to store data in a dynamic fashion in a layout on the persistent storage devices of the storage entity that meets the different data requirements for the data while still being within the constraints for the persistent storage devices. The storage entity is configured to change the layout for a portion of the data as requirements related to at least one of performance or resiliency for a portion of the data change while the storage entity continues to provide the data from the storage entity.

Live configurable storage

A system in which a file system may operate on a volume in which the logical address extent of the volume is divided into multiple tiers, each tier providing storage having a distinct trait set by mapping the logical addresses of the volume to appropriate underlying storage systems. A volume system exposes the volume to the file system in a manner that the file system itself has awareness of the tiers, and is aware of the trait sets of each tier. The file system may thus store file system namespaces (such as directories and files) into the tiers as appropriate for the file system namespace. A provisioning system may also be provided and be configured to provision the volume to include such tiers, and if desired, to extend the tiers.

Heterogenic volume generation and use system

A thinly provisioned storage system detects whether physical storage capacity is available when there is a request to allocate storage capacity, prior to data being written to the storage system. In particular, at the time when the file system allocates storage, such as when creating a file or performing an extending write (append) operation, allocating storage to an unallocated region of a sparse file, defragmenting a file, and the like, a storage system can verify that actual physical storage capacity is available. Thus, if there is insufficient actual physical capacity at the time when a storage allocation is attempted, then an error message can be sent and remedial action can be taken.

Managing capacity of a thinly provisioned storage system

Embodiments are directed to creating global, aggregated namespaces for storage management and to providing consistent namespaces in a distributed storage system. In one scenario, a computer system defines data storage objects for each data storage node. The data storage objects uniquely identify storage elements of the data storage nodes, where each data storage object includes various associated attributes. The computer system replicates the defined data storage objects and any associated attributes from a first data storage node to a second, different data storage node among the data storage nodes. As such, the defined data storage objects are visible from any node in the data storage nodes. The computer system also aggregates the defined data storage objects for each of the data storage nodes and creates a global, aggregated namespace that includes the aggregated data storage objects for each of the data storage nodes.

Creating global aggregated namespaces for storage management

The provisioning of a volume that has multiple tiers corresponding to different trait sets. The volume to be provisioned is identified along with multiple tiers that are to be in the volume. For each of the tiers that are to be provisioned within the volume, a corresponding trait set is identified as to be applied to each tier. This corresponding trait set may be based on underlying storage systems that are available at the time of provisioning, or which are anticipated to be available. The volume is then caused to be provisioned with the corresponding tiers having the corresponding trait sets. Also, the provisioning of a file, which is determined to have one or more storage traits. Based on these storage traits, the file is then caused to be assigned to an appropriate tier.

Provisioning in heterogenic volume of multiple tiers

The storage of data sets in a storage set (e.g., data sets written to hard disk drives comprising a RAID array) may diminish the performance of the storage set through non-sequential writes, particularly if the storage devices promptly write data sets that are followed by sequentially following data sets. Additionally, storage sets may exhibit inconsistencies due to non-atomic writes of data sets and verifiers (e.g., checksums) and an intervening failure, such as an occurrence of the RAID write hole. Instead, data sets and verifiers may first be written to a stored on the nonvolatile media of a storage device before being committed to the storage set. Such writes may be sequentially written to the journal, irrespective of the locations of the data sets in the storage set; and recovery of a failure may simply involve re-committing the consistent records in the journal to correct incomplete writes to the storage set.

Nonvolatile media journaling of verified data sets

A volume system that presents a volume having an extent of logical addresses to a file system. A volume exposure system exposes the volume to the file system in a manner that the volume has multiple tiers, each offering storage of different traits. This is performed using multiple heterogenic underlying storage systems, each having different storage system-specific traits. Each underlying storage system may be hardware, software, or a combination thereof that permits each storage system to expose storage having the particular storage system-specific traits to the file system. The volume system supports each tier by mapping logical addresses of the tier to portions of underling storage systems that are consistent with the tier traits.

Volume having tiers of different storage traits

Techniques for recovery and redistribution of data from a virtual disk storage system are described herein. In one or more implementations, a storage scheme derived for a virtual disk configuration is configured to implement various recovery and redistribution designed to improve recovery performance. The storage scheme implements one or more allocation techniques to produce substantially uniform or nearly uniform distributions of data across physical storage devices associated with a virtual disk. The allocation facilitates concurrent regeneration and rebalancing operations for recovery of data in the event of failures. Additionally, the storage scheme is configured to implements parallelization techniques to perform the concurrent operations including but not limited to controlling multiple parallel read/writes during recovery.

Virtual disk recovery and redistribution

The representation of storage devices on computers (e.g., as logical volumes) may be complicated by the pooling of multiple storage devices in order to apply redundancy plans such as mirroring and checksumming. Presented herein is a storage device driver configured to operate as a storage device interface generating representations of the storage regions of the storage devices; to claim those regions as a storage controller; and to expose pooled storage regions as logical disks. Additionally, the storage device driver may support the inclusion of storage devices in a cluster, comprising nodes that may be appointed as managers of the storage pool configuration; as managers of the storage devices; as owners having exclusive read/write access to the storage pool or cluster resources; and as cluster resource writers having exclusive write access to a cluster resource. The nodes of the cluster may interoperate to share the storage devices while avoiding write conflicts.

Storage device drivers and cluster participation

An integrity unit can be calculated from a first data unit, and a first storage device can be requested to store the first data unit. A second storage device, which can be separate from and/or a different type of device from the first storage device, can be requested to store metadata, which includes the integrity unit, in nonvolatile memory. Also, a second data unit can be received from the first storage device in response to a request for the first data unit. The integrity unit can be received from the second storage device, and the second data unit and the integrity unit can be analyzed to determine whether the second data unit matches the first data unit. Alternatively, a first integrity unit can be stored in a metadata region of a nonvolatile memory block, where the block also stores the data from which the first integrity unit was calculated.

Data integrity units in nonvolatile memory

A multi-device storage system can be arranged into power saving systems by placing one or more storage devices into a reduced power consuming state when the storage activity associated with the system is sufficiently reduced that an attendant decrease in throughput will not materially affect users of the storage system. Where data redundancy is provided for, a redundant storage device can be placed into the reduced power consuming state and its redundancy responsibilities can be transitioned to a partition of a larger storage device. Such transitions can be based on specific parameters, such as write cycles or latency, crossing thresholds, including upper and lower thresholds, they can also be based on pre-set times, or a combination thereof. Lifecycle information, including lifecycle information collected in real-time by storage devices on a block-by-block basis, can be utilized to obtain historical empirical data from which to select the pre-set times.

Growing community of inventors

Sammamish, WA, United States of America

Shiv Kumar Rajpal