Emc IP Holding Company LLC

Emc Corporation

Los Alamos National Security, LLC

Triad National Security, LLC

Ut-battelle, Inc.

Emc IP Holding Corporation LLC

Sorin Faibish

John Michael Bent

James M Pedone, Jr

Philippe Armangau

Uday K Gupta

Percy Tzelnic

Istvan Gonczi

Ivan Bassov

Gary Alan Grider

Xiaoye Jiang

Peter C Bixby

Aaron Torres

Wayne William Duso

John Forecast

Vamsi K Vankamamidi

Stephen A Fridella

Dennis P Ting

Ivan Basov

Dennis P J Ting

Sassan Teymouri

Philip N Shilane

Zhenhua Zhang

Jason Glasgow

Tao Peng

James J Moore

Seyfullah H Oguz

Daniel Gardere

Dominique P Cote

Jingwang Zhang

Assaf Natanzon

Per W Brashers

Rustem Rafikov

Gang Ma

Michel Noury

Xuezhao Liu

Christopher Alan Seibel

Ugur Sezer

Eriks Sasha Paegle

Patrick S Combes

James Pedone

Haiying Tang

Jiannan Zheng

Eyal Zimran

Haiyun Bao

Stephen W Poole

Dennis Pei Jean Ting

Anton Kucherov

Jeffrey Mark Nick

Yedidya Dotan

Walter A O'Brien, Iii

Philip Derbeko

William C Whitney

Sachin Mullick

Rui Liang

David Bonnie

Hsing-bung Chen

Sebastian Keller

Jie Wang

Sasan Teymouri

Jun He

Jean-Pierre Bono

Roy E Clark

Lawrence N Friedman

Sairam Veeraswamy

Daniel E Cummins

Yael Villa

Sitaram D Pawar

Jeremy O'Hare

Srinivasa Rao Vempati

Christopher H Stacey

Mark K Ku

Robert S Philpott

John F Gillono

Matthew H Fredette

Yuri Manusov

Sen Zhang

James Theodore Compton

Samuel Adams

Joshua E Simons

Gerald Cotter

John P Didier

Arthur Otis Harris

Mario Wurzl

Richard G Chandler

Alexandre Lemay

John Alexander Murray

George Tuma

Adam Manzanares

Ronald A Miller, Ii

Gurjeet S Arora

Yehoshoua Sasson

William O Hultin

Sebastien Keller

Raymond Mantchala

Jean Louis Rochette

Caiping Zheng

Dennis Ting, Jr

Bradley W Settlemyer

Boyd Wilson

Naomi Eskira

A technique for performing data deduplication operates at sub-block granularity by searching a deduplication database for a match between a candidate sub-block of a candidate block and a target sub-block of a previously-stored target block. When a match is found, the technique identifies a duplicate range shared between the candidate block and the target block and effects persistent storage of the duplicate range by configuring mapping metadata of the candidate block so that it points to the duplicate range in the target block.

Deduplicating data at sub-block granularity

A method is used in analyzing a storage system using a machine learning system. Data gathered from information associated with operations performed in a storage system is analyzed. The storage system is comprised of a plurality of components. A bitmap image is created based on the gathered data, where at least one of the plurality of components is represented in the bitmap image. The machine learning system is trained using the bitmap image, where the bitmap image is organized to depict the plurality of components of the storage system.

Analyzing storage systems using machine learning systems

A method, computer program product, and computer system for identifying duplicate sectors in a block of a plurality of blocks. The duplicate sectors in the block may be zeroed out. A data reduction operation may be performed on the block after the duplicate sectors are zeroed out.

System and method for sub-block deduplication with search for identical sectors inside a candidate block

Techniques for detecting state changes in a system may include receiving a first neural network that is trained to detect when the system transitions into a first resulting state, wherein the system transitions into at least a first intermediate state prior to transitioning into the final resulting state; training the first neural network using a first plurality of inputs denoting the system in the first intermediate state; obtaining a plurality of sets of internal state information of the first neural network, each set of the plurality of sets denoting an internal state of the first neural network at a different point in time after the first neural network has processed at least a portion of the first plurality of inputs; and training a second neural network, using the plurality of sets of internal state information, to detect the first intermediate state.

Using machine learning to detect system changes

A method system, and computer program product for improving data reduction using aggregate machine learning systems comprising receiving, by an aggregating machine learning system from one or more machine learning systems associated with a set of one or more storage arrays, a first set of output parameters indicative of performance metrics for the set of the one or more storage arrays, aggregating, by the aggregating machine learning system, the first set of output parameters, resulting in a second set of output parameters, and sending, from the aggregating machine learning system, at least one member of the second set of output parameters as an input to at least one of the one or more machine learning systems associated with the set of the one or more storage arrays.

Data reduction improvement using aggregated machine learning

A method performed by a block-storage server, of storing data is described. The method includes (1) receiving, from a remote file server, data blocks to be written to persistent block storage managed by the block-storage server; (2) receiving, from the remote file server, metadata describing files to which the data blocks belong in a set of filesystems managed by the remote file server; and (3) selectively applying data reduction when storing the data blocks in the persistent block storage based, at least in part, on the received metadata. An apparatus, system, and computer program product for performing a similar method are also provided.

File layer to block layer communication for selective data reduction

A method, computer program product, and computer system for identifying a plurality of blocks. At least one heuristic associated with at least a portion of the plurality of blocks may be determined. It may be determined whether to compress at least the portion of the plurality of blocks based upon, at least in part, the at least one heuristic. At least the portion of the plurality of blocks may be compressed based upon, at least in part, the at least one heuristic.

System and method for improving space efficiency by compressing multi-block aggregates

Continuous learning may include receiving a first neural network trained using a first training data set to predict outputs; determining whether the first neural network has a successful prediction rate greater than a prediction threshold; and responsive to determining the first neural network does not have a successful prediction rate greater than the prediction threshold, performing processing. The processing may include training the first neural network using a second training data set different than the first training data set; determining that a trigger condition has occurred, wherein the trigger condition includes detecting, during training of the first neural network using the second training data set, that a first weight of the first neural network has a corresponding weight change exceeding a threshold weight change; responsive to determining the trigger condition has occurred, reconfiguring the first neural network; and training the reconfigured first neural network using the second training data set.

Continuous learning techniques with machine learning systems to adapt to new input data sets

A method performed by a block-storage server, of storing data is described. The method includes (1) receiving, from a remote file server, data blocks to be written to persistent block storage managed by the block-storage server; (2) receiving, from the remote file server, metadata describing a placement of the data blocks in a filesystem managed by the remote file server; and (3) organizing the data blocks within the persistent block storage based, at least in part, on the received metadata. An apparatus, system, and computer program product for performing a similar method are also provided.

File layer to block layer communication for block organization in storage

A technique for managing data storage includes generating entropy of blocks on a per-block basis and selectively performing inline compression on blocks based at least in part on their entropy. Entropy of a block provides a rough measure of the block's compressibility. Thus, using per-block entropy enables a storage system to steer compression decisions, e.g., whether to compress and/or how much to compress, flexibly and with high granularity, striking a balance between throughput and storage efficiency.

Adaptive inline compression

A technique for performing data reduction applies deduplication principles when performing data compression, providing a form of enhanced compression. The technique obtains a chunk of data that contains multiple extents and applies deduplication actions to identify duplicate extents within the chunk. The technique marks duplicate extents in metadata. Such duplicate extents need not be compressed using conventional data compression, saving computational resources and considerable time.

Compressing data using deduplication-like methods

Techniques for generating data sets may include: receiving an initial buffer that achieves a compression ratio responsive to compression processing using a compression algorithm, the initial buffer including first content located at a first position in the initial buffer and including second content located at a second position in the initial buffer; and generating a data set of buffers using the initial buffer. The data set may be expected to achieve a specified deduplication ratio responsive to deduplication processing and to achieve the compression ratio responsive to compression processing using the compression algorithm. Generating the data set may include generating a first plurality of buffers where each buffer of the first plurality is not a duplicate of another buffer in the first plurality, and generating a second plurality of duplicate buffers. Each duplicate buffer may be a duplicate of a buffer in the first plurality of buffers.

Techniques for generating data sets with specified compression and deduplication ratios

A method, computer program product, and computing system for receiving a candidate data portion; calculating a distance-preserving hash for the candidate data portion; and performing an entropy analysis on the distance-preserving hash to generate a hash entropy for the candidate data portion.

System and method for hash-based entropy calculation

A method, computer program product, and computer system for identifying a plurality of blocks. At least one heuristic associated with at least a portion of the plurality of blocks may be determined. It may be determined whether at least the portion of the plurality of blocks is a candidate for deduplication based upon, at least in part, the at least one heuristic. At least the portion of the plurality of blocks may be deduplicated based upon, at least in part, the at least one heuristic.

System and method for detecting deduplication opportunities

A method is used in managing data reduction in storage systems using machine learning. A value representing a data reduction assessment for a first data block in a storage system is calculated using a hash of the data block. The value is used to train a machine learning system to assess data reduction associated with a second data block in the storage system without performing the data reduction on the second data block, where assessing data reduction associated with the second data block indicates a probability as to whether the second data block can be reduced.

Managing data reduction in storage systems using machine learning

Growing community of inventors

Newton, MA, United States of America

Sorin Faibish