Emc IP Holding Company LLC

Maginatics LLC

Dropbox, Inc.

Rajiv Desai

Nathan Rosenblum

Thomas Manville

Julio Lopez

Niraj Tolia

Vaibhav Kamra

Shrinand Javadekar

Vijay Panghal

Bashar Al-Rawi

Shashank Senapaty

Peng Kang

Daisy Luo Zhou

A write request is transmitted from a client to a metadata server ('MDS'), wherein the write request comprises an object identifier associated with a data object. An object store location is received for an object store from the MDS. A metadata request is transmitted to the object store using the object store location, wherein the metadata request includes the object identifier. A metadata response is received from the object store. Determine the metadata response contains an object designator. A count associated with a mapping between the object identifier and the object designator is incremented, wherein the mapping resides on an object version manager shared with a second MDS.

Global data deduplication across multiple distributed file systems

A write request is transmitted from a client to a metadata server ('MDS'), wherein the write request comprises an object identifier associated with a data object. An object store location for an object store is received from the MDS. A conditional write request is transmitted to the object store using the object store location, wherein the conditional write request includes the object identifier and the data object. A metadata response is received from the object store. The metadata response is determined to contain an object designator. A commit request is transmitted to the MDS, wherein the commit response includes the object designator.

Data deduplication on a distributed file system using conditional writes

A method, article of manufacture, and apparatus for limited deduplication scope on a distributed file system is discussed. A write request is received from a client at the metadata server ('MDS'), where the write request comprises a data object identifier and a preferred object store identifier. The MDS determines whether a preferred object store associated with the preferred object store identifier contains a copy of a data object associated with the data object identifier. A write URL comprising the data object identifier and a object store location associated with the preferred object store is transmitted to the client when the preferred object store does not contain the copy of the data object.

Limited deduplication scope for distributed file systems

A combined object associated with a data chunk included in a request file is determined. An encryption key associated with the combined object and a corresponding chunk hash value associated with the data chunk are used to determine a corresponding chunk key associated with the data chunk. At least a locator to be used to retrieve the combined object and the corresponding chunk key associated with the data chunk are provided to a requesting system.

Secure cloud-based storage of data shared across file system objects and clients

Techniques to provide a de-duplicating distributed file system using a cloud-based object store are disclosed. In various embodiments, a request to store a file comprising a plurality of chunks of file data is received. A determination to store at least a subset of the plurality of chunks is made. The request is responded to at least in part by providing an indication to store two or more chunks comprising the at least a subset of the plurality of chunks comprising the file as a single stored object that includes the combined chunk data of said two or more chunks.

De-duplicating distributed file system using cloud-based object store

A method, article of manufacture, and apparatus for providing a write leases in a distributed file system is discussed. A lease break is received for a file at a client from a metadata server ('MDS'), wherein the client has a write lease for the file. A write buffer is flushed on the client to an object store, wherein the write buffer comprises data objects comprising the file. A metadata request is transmitted from the client to the MDS in connection with flushing the write buffer. A lease extension is received for the write lease from the MDS.

Write leases for distributed file systems

Techniques to provide access to file system information are disclosed. In various embodiments, an indication that a user input associated with creating a local copy of a file system object has been received is received at a file system client. The file system client determines that a user-driven pre-fetch option has been set with respect to the file system object. The file system client pre-fetches content data associated with the file system object, based at least in part on the determination that the pre-fetch option has been set with respect to the file system object.

User driven data pre-fetch

A method, article of manufacture, and apparatus for deduplicating data on a distributed file system is discussed. Metadata requests are made to an object store to determine whether the store already contains a data object. If the object store does contain the data object, a commit request is sent to a metadata server. If the object store does not contain the data object, the data object is transmitted to the object store.

Data deduplication on a distributed file system

A method, article of manufacture, and apparatus for accessing a distributed file system using a content delivery network is discussed. An IO request for an object is received at a metadata server comprising metadata mapping a plurality of objects to an object store. An object identifier associated with the object and a CDN domain associated with the object store are determined. A URL is generated from the object identifier and the CDN domain. The URL is then returned to the client.

Distributed file systems on content delivery networks

Techniques to provide secure cloud-based storage of data shared across file system objects and clients are disclosed. In various embodiments, a primary encryption key is determined for an object associated with a plurality of component chunks of file system data. The primary encryption key is used to generate for each of said component chunks a corresponding chunk key, based at least in part on the primary encryption key and data comprising or otherwise associated with the chunk. The respective chunk keys are provided to a file system client configured to create and store the object at least in part by encrypting each chunk included in the plurality of component chunks using the chunk key provided for that chunk to generated encrypted chunk data, and combining the encrypted chunk data to create and store the object.

A write request is transmitted from a client to a metadata server ('MDS'), wherein the write request comprises an object identifier associated with a data object. An object store location is received for an object store from the MDS. A metadata request is transmitted to the object store using the object store location, wherein the metadata request includes the object identifier. A metadata response is received from the object store. Determine the metadata response contains an object designator. A count associated with a mapping between the object identifier and the object designator is incremeneted, wherein the mapping resides on an object version manager shared with a second MDS.

A method, article of manufacture, and apparatus for providing a site cache in a distributed file system is discussed. Data objects may be written to a site cache prior to writing them to an authoritative object store. This provides performance benefits when a client writing the data has a better connection to the site cache than to the authoritative object store.

Site cache for a distributed file system

A large-scale asynchronous processor of events in a distributed and parallel processing environment. The processor includes a plurality of processes executing on a plurality of interconnected processors. The plurality of processes including a frontend process, a plurality of dispatcher processes, a plurality of job queues, and a plurality of worker processes. The frontend process is for receiving event notification remote procedure calls from one or more event sources and for publishing event notifications, based on the received calls, to topics of a distributed streaming platform. The plurality of dispatcher process are for consuming event notifications from the topics and for en-queuing jobs, based on the consumed event notifications, to the plurality of job queues. The plurality of worker process are for pulling jobs from the plurality of job queues and for the processing the jobs.

Growing community of inventors

Mountain View, CA, United States of America

Rajiv Desai