Microsoft Technology Licensing, LLC

Microsoft Corporation

Ran Kalach

Paul Adrian Oltean

James Robert Benton

Mathew James Dickson

Ravisankar V Pudipeddi

Kashif Hasan

Jin Li

Abhishek Gupta

Sudipta Sengupta

Chun Ho Cheung

Neal Robert Christiansen

Sarosh Cyrus Havewala

Ravinder S Thind

Nir Ben-Zvi

Michael L Michael

Jason Daniel Shay

Ronakkumar N Desai

Ahmed El-Shimi

Spencer Alan Bishop

Matthias Hermann Wollnik

Cristian Teodorescu

Paul Brandon Luber

Clyde Law

Abhinav Srivastava

William L Scheidel

Jingbo Wu

Dinesh Haridas

Fenghua Yuan

Georgi M Matev

Chun Ho (Ian) Cheung

Ramarathnam R Venkatesan

Robert M Fries

Norbert Paul Kusters

Sanjeev Mehrotra

Surendra Verma

Brian Thomas Berkowitz

Roy Peter D'Souza

James Ernest Johnson

Catharine Van Ingen

Sriram Sampath

Anders Samuelsson

Jonathan M Cargille

Rajeev Y Nagar

Reuven Lax

Ping Xie

Omkant Pandey

Michael S Murstein

Jeffrey B Hamblin

Dana D Groff

Nikhil Vijay Chandhok

Raja Pazhanivel Perumal

Manoj K Valiyaparambil

Ziquan Li

Joerg-Thomas Pfenning

Daniel Hefenbrock

Giridharan Kasirala Ramachandraiah

Jeremiah J Moon

P Adrian Oltean

Paul B Luber

Cristian G Teodorescu

Techniques are described herein that are capable of heterogeneously optimizing a file. Heterogeneous optimization involves optimizing regions of a file non-uniformly. For example, the regions of the file may be optimized to different extents. In accordance with this example, a different optimization technique may be used to optimize each region or subset of the regions. In one aspect, optimization designations are assigned to respective regions of a file based on access patterns that are associated with the respective regions. The file may be a database file, a virtualized storage file, or other suitable type of file. Each optimization designation indicates an extent to which the respective region is to be optimized. Each region may be optimized to the extent that is indicated by the respective optimization designation that is assigned to that region.

Heterogeneous file optimization

A system hosts a file system representing a replica of file systems hosted on a plurality of nodes, each node storing data on a first storage medium. The system stores data on a second storage medium that is slower than the first storage medium. The system stores on the second storage medium a file received from one of the nodes when the file remains un-accessed by the nodes for a predetermined period of time. The first storage medium on the plurality of nodes stores a pointer to the file and not the data content of the file stored on the second storage medium. The system provides the file from the second storage medium to the first storage medium on one of the nodes while providing the plurality of nodes read and write access to the file stored on the second storage medium using the pointer on the first storage medium.

Reliable hierarchical storage management with data synchronization

A system includes determination of a resource stored in a cloud storage service, identification of a first cloud service associated with the resource, determination of a first pre-backup callback and a first post-backup callback associated with the first cloud service, transmission of the first pre-backup callback to the first cloud service, reception of a return of the first pre-backup callback from the first cloud service, transmission, in response to receipt of the return, of an instruction to the cloud storage service to backup the resource, and transmission of the first post-backup callback to the first cloud service.

Cloud services integrated backup and restore

Data streams may be stored in a chunk store in the form of stream maps and data chunks. Data chunks corresponding to a data stream may be stored in a chunk container, and a stream map corresponding to the data stream may point to the data chunks in the chunk container. Multiple stream maps may be stored in a stream container, and may point to the data chunks in the chunk container in a manner that duplicate data chunks are not present. Techniques are provided herein for localizing the storage of related data chunks in such chunk containers, for locating data chunks stored in chunk containers, for storing data streams in chunk stores in localized manners that enhance locality and decrease defragmentation, and for reorganizing stored data streams in chunks stores.

Scalable chunk store for data deduplication

The subject disclosure is directed towards partially recalling file ranges of deduplicated files based on tracking dirty (write modified) ranges (user writes) in a way that eliminates or minimizes reading and writing already-optimized adjacent data. The granularity of the ranges does not depend on any file-system granularity for tracking ranges. In one aspect, lazy flushing of tracking data that preserves data-integrity and crash-consistency is provided. In one aspect, also described is supporting granular partial recall on an open file while a data deduplication system is optimizing that file.

Granular partial recall of deduplicated files

Techniques are described herein that are capable of optimizing (i.e., deduplicating) data in a virtualization environment. For example, optimization designations (a.k.a. deduplication designations) may be assigned to respective regions of a virtualized storage file. A virtualized storage file is a file that is configured to be mounted as a disk or a volume to provide a file system interface for accessing hosted files. In accordance with this example, each optimization designation indicates an extent to which the respective region is to be optimized (i.e., deduplicated). In another example, a virtualized storage file is mounted to provide a virtual disk that includes hosted files. In accordance with this example, optimization designations are assigned to the respective hosted files. In further accordance with this example, each optimization designation indicates an extent to which the respective hosted file is to be optimized.

Data deduplication in a virtualization environment

Disclosed herein is a system and method that can be used to synchronize files from a hosted file system where the client endpoint devices have direct access to the files stored on a hosted file store/system. When the client synchronizes with the files stored on the hosted file store the synchronization process identifies the files that need to be synchronized with the client and prepares batches of files to be synced. During the process of synchronization the files are not locked from other user's being able to use the files. If the file is modified by another endpoint during the file synchronization process the file will not be synchronized with the client device during the particular synchronization process. In this way synchronization becomes possible with the hosted service without impacting the performance of other devices that use and access the data in the hosted file store.

Hosted file sync with direct access to hosted files

Techniques for backup and restore of optimized data streams are described. A chunk store includes each optimized data stream as a plurality of chunks including at least one data chunk and corresponding optimized stream metadata. The chunk store includes data chunks in a deduplicated manner. Optimized data streams stored in the chunk store are identified for backup. At least a portion of the chunk store is stored in backup storage according to an optimized backup technique, an un-optimized backup technique, an item level backup technique, or a data chunk identifier backup technique. Optimized data streams stored in the backup storage may be restored. A file reconstructor includes a callback module that generates calls to a restore application to request optimized stream metadata and any referenced data chunks from the backup storage. The file reconstructor reconstructs the data streams from the referenced data chunks.

Backup and restore strategies for data deduplication

The subject disclosure is directed towards a data deduplication technology in which a hash index service's index is partitioned into subspace indexes, with less than the entire hash index service's index cached to save memory. The subspace index is accessed to determine whether a data chunk already exists or needs to be indexed and stored. The index may be divided into subspaces based on criteria associated with the data to index, such as file type, data type, time of last usage, and so on. Also described is subspace reconciliation, in which duplicate entries in subspaces are detected so as to remove entries and chunks from the deduplication system. Subspace reconciliation may be performed at off-peak time, when more system resources are available, and may be interrupted if resources are needed. Subspaces to reconcile may be based on similarity, including via similarity of signatures that each compactly represents the subspace's hashes.

Using index partitioning and reconciliation for data deduplication

Described is caching classification-related metadata for a file in an alternate data stream of that file. When a file is classified (e.g., for data management), the classification properties are cached in association with the file, along with classification-related metadata that indicates the state of the file at the time of caching. The classification-related metadata in the alternate data stream is then useable in determining whether the classification properties are valid and up-to-date when next accessed, or whether the file needs to be reclassified. If the properties are valid and up-to-date, they may be used without requiring the computationally costly steps of reclassification. Also described is using more than one alternate data stream for the cache, and extending the classification-related metadata through a defined extension mechanism.

Alternate data stream cache for file classification

A host server hosting one or more virtual machines can back up host volumes and the one or more virtual machines installed thereon in an application-consistent manner. In one implementation, a host-level requestor instructs a host-level writer to identify which virtual machines qualify for application-consistent backups. The host-level requestor then instructs the host-level writer to initiate virtual machine backups through guest-level requesters in each appropriately-configured virtual machine, wherein the virtual machines create application-consistent backups within the virtual machine volumes. The host-level requester then initiates snapshots of the server volumes on the host-level. The virtual machine-level snapshots can thus be retrieved from within the host-level snapshots of the server volumes.

Creating host-level application-consistent backups of virtual machines

The subject disclosure is directed towards encryption and deduplication integration between computing devices and a network resource. Files are partitioned into data blocks and deduplicated via removal of duplicate data blocks. Using multiple cryptographic keys, each data block is encrypted and stored at the network resource but can only be decrypted by an authorized user, such as domain entity having an appropriate deduplication domain-based cryptographic key. Another cryptographic key referred to as a content-derived cryptographic key ensures that duplicate data blocks encrypt to substantially equivalent encrypted data.

Integrated data deduplication and encryption

Growing community of inventors

Bellevue, WA, United States of America

Ran Kalach