Emc Corporation

Christopher H Stacey

Eyal Zimran

Edward C Bueche

Peter W Madany

Daniel S Collins

Sunita Casula

Oluwasoga T Oni

Umer Toor

Allen B Hubbe

Navaneeth Ranganna

Suvidya S Daftardar

Richard P Ruef

Ivan Basov

Christian J Chuba

Sorin Faibish

Jean-Pierre Bono

Uday K Gupta

Xiaoye Jiang

John T Cardente

Peter C Bixby

Stephen A Fridella

Morgan A Clark

John Michael Hayden

Jiannan Zheng

Paul LeMahieu

William C Whitney

James Theodore Compton

Stephen A Rago

Jerald W Gross

Augustine Amegadzie

Mary Walker

A method of searching shared files stored on a shared storage includes providing a search functionality in the shared storage having a set of search indexes to perform a search based on a search term; providing an event stream of update events associated with the shared files in the shared storage; processing the event stream to generate a processed event stream; and responding to update events in the processed event stream by indexing files referenced in the update events in the processed event stream where the indexing is performed to maintain a predetermined level of quality of service at the shared storage.

Integrated search for shared storage using index throttling to maintain quality of service

An improved technique involves proactively adjusting data storage system configuration in response to detecting external events. The improved technique predicts load changes based on historical correlations between events and load changes and directs an administrator to modify system configurations to prepare for the predicted changes in load. Advantageously, the improved technique enables administrators to better prepare for changes in load brought about by external events and thus to better maintain required service levels. Further, the improved technique reduces need for stressful and urgent responses by system administrators.

Dynamic event driven storage system optimization

Directory attributes are provided so that the time for creating an incremental backup of a file system by a scan of the file system tree is proportional generally to the number of files that change between backups instead of the number of files in the file system. A tree modification attribute indicates whether or not any file in a directory tree has changed since the last backup. If no file has changed in the directory tree, then the entire tree is skipped during the scan for changed files. In addition, a list is compiled of the files in the directory that represent branches having at least one changed file, so that the list is scanned instead of the directory entries. When a file is changed for the first time since the last backup, the file is queued so that the directory attributes are updated in a background process.

Rapid incremental backup of changed files in a file system

Needless migration of duplicate file data is avoided by recording in a database a record of migration of a first instance of a replicated file from a source file server to a target file server. When migrating a subsequent instance of a replicated file to the target file server, the database is searched for a record of replication of a first instance of the replicated file to the target file server. If such a record is found, the subsequent instance is replicated by sending from the source file server to the target file server attributes of the subsequent instance of the replicated file and an identification of the previously migrated data of the replicated file without re-transmission of the previously migrated data of the replicated file.

Efficient migration of replicated files from a file server having a file de-duplication facility

Implementations are provided herein relating to augmenting metadata collection within a storage platform. The storage platform can be audited to determine the types of metadata currently being gathered within the storage platform, and the schedule for when that information is gathered. The storage platform can receive a request to generate metadata, compare the requested information with the previously generated and/or scheduled generation of metadata. Rather than redundantly gathering the same metadata via multiple requests, known metadata or scheduled retrieval of known metadata can be used to process portions of the metadata request, and any metadata that was not previously generated can then be separately generated. In this sense, the metadata collection within a storage platform can be augmented to gather additional metadata requested outside the storage platform in an efficient matter that does not unnecessarily increase scanning activity within the storage platform.

Augmenting metadata collection for stored data analytics

A file server system having solid-state drive primary storage and magnetic disk drive secondary storage aggressively moves newly created files from the primary storage to selected file systems in the secondary storage to match expected access patterns upon the files to different configurations of the file systems and for load balancing upon the file systems in the secondary storage. Upon read access to a file that has been moved to the secondary storage, or upon migration of a newly created file that was read in primary storage after creation, a corresponding stub file containing file mapping metadata is created in the primary storage. The file mapping metadata in the stub file maps the extent of the file to logical storage addresses in the secondary storage.

File server system having tiered storage including solid-state drive primary storage and magnetic disk drive secondary storage

A method of searching shared files stored on a general purpose shared storage includes receiving a search term in a native search interface of an operating system on a computing device; and performing, on the computing device, a search based on the search term including accessing a search functionality of the general purpose shared storage, wherein the search functionality uses indexes of the shared files stored on the shared storage and updates the indexes of the shared files in response to update events generated from the shared files. In one embodiment, the method performs a federated search based on the search term.

Integrated search for shared storage

A file server having a redundant data elimination store responds to requests from clients for copies of space-reduced files. For reduced storage requirements and reduced processing time, a baseline version of unshared data blocks is created in the file server, and then the baseline version is space reduced by conversion to a stub version of shared data blocks, and then the file server responds to a request from a client for creation of one or more space-reduced copies by cloning the stub version. The file server may also respond to a request for creation of a modified space-reduced copy having specified data written at a specified offset. For example, the file server creates a respective modified copy of a virtual server image file for use by each client of the file server.

Creating copies of space-reduced files in a file server having a redundant data elimination store

A method, article of manufacture, and apparatus for efficiently backing up information are disclosed. In some embodiments, this comprises receiving data and analyzing the data. In some embodiments, the data may be manipulated. In some embodiments, the data may come from multiple sources. Chunk headers, data chunks and chunk maps are then created based on analyzing the data. Chunk headers store the data manipulation technique of the data chunk. In some embodiments, chunk headers store the size of the manipulated and un-manipulated data chunks. The chunk headers, data chunks and chunk maps are then organized into a datastream. The datastream is then stored in a storage device. In some embodiments, the received data may be discarded. In some embodiments, a chunk header is associated with a data chunk.

Time and bandwidth efficient backups of space reduced data

Accordingly a method and interface allows an attribute data base used by an Information Manager to be quickly populated and accurately maintained. A single Bulk Attribute Retrieval Request triggers the primary storage device to collect object attribute information. The method allows for selective collection of objects and attributes by providing filters and attribute lists in the Requests. The Request may be used to provide an incremental scan with appropriate time stamp filtering. In addition, the size of the results can be controlled by the IM by eliminating attributes that are not of interest to the IM. The Request is advantageously issued over a FileMover interface, which is an HTTP connection, and encoded in XML, allowing the IM to easily customize the Request as desired.

Method and apparatus for performing bulk file system attribute retrieval

A method, article of manufacture, and apparatus for efficiently recovering information are disclosed. In some embodiments, this comprises receiving a datastream and analyzing the datastream for chunk headers, data chunks and chunk maps. The chunk headers are then used to determine the data manipulation technique of the data chunks. In some embodiments, the chunk headers contain information about the size of the manipulated and un-manipulated data chunks. Data is then reconstructed from the datastream analysis and data manipulation technique determination. The reconstructed data is then stored in a computer readable medium. In some embodiments, the reconstructed data may be stored in multiple destinations. In some embodiments, the storage of the reconstructed data in multiple destinations may be simultaneous.

Time and bandwidth efficient recoveries of space reduced data

A protected file is retained in secondary storage for a retention period, and a corresponding stub file is retained in primary storage for the retention period. The stub file retains attributes of the file indicating a location of the file data in the secondary storage and indicting that the stub file is to be retained for the retention period. A system administrator may instruct a primary file server that a secondary file server supports retention based protection and it is desired to protect stub files corresponding to protected files that are migrated to the secondary file server. The primary file server may request the secondary file server to return a protection status of the file after migration. This enables automatic policy-based migration to both file level retention (FLR) and non-FLR retention protected secondary storage from diverse applications accessing FLR or non-FLR files in the primary storage.

Primary stub file retention and secondary retention coordination in a hierarchical storage system

When a directory tree is migrated from a source file system to a target file system, the directories of the directory tree are retained in the source file system, and handle translation entries are substituted for the regular data files of the directory tree in the source file system. The handle translation entries provide new file handles for the regular data files when the source file system is accessed with old file handles of the regular data files. For example, each handle translation entry includes the old file handle, a file system identifier, and the new file handle. The old file handle provides verification of the file handle being used for accessing the handle translation entry, and the file system identifier identifies the target file system in which the new file handle is valid.

File handle banking to provide non-disruptive migration of files

A hierarchical storage system is constructed from file servers and a policy engine server by building upon a file migration service. Offline attributes are added to file system inodes in a primary file server, file system parameters are added in the primary server, an offline read access method field is added to a connection database, and the primary file server is programmed to use these attributes and parameters for selecting a particular method (such as a partial read migration or a pass through read method) for read access to an offline file. In this fashion, the primary file server is provided with flexibility for selecting a particular one of a number of migration methods depending on various conditions, in order to improve the trade-off between the cost of storage and delay in file access.

Growing community of inventors

Chistchurch, New Zealand

Christopher H Stacey