Western Digital Technologies, Inc.

Stijn Blyweert

Koen De Keyser

Frederik Jacqueline Luc De Schrijver

Mark Christiaens

Techniques are described for reducing power consumption in a distributed data storage system using a hierarchy rule that is generated based on a spreading policy and a set of tolerable failures. A method may operate to distribute erasure-encoded data of a first data object across first and second portions of a distributed storage system using a hierarchy rule corresponding to a spreading policy based on a set of tolerable failures from which the first data object can be recovered. The method disables the first portion of the distributed storage system that includes a first portion of the erasure-encoded data. The first portion of the distributed storage system is determined according to the spreading policy and the hierarchy rule identifies the set of tolerable failures.

Power reduction in distributed storage systems

A data object can be encoded into a plurality of encoded data fragments and stored on backend storage elements in a distributed encoded storage system. The identifiers and metadata corresponding to each encoded fragment of the data object can be stored in a single metadata unit, which is stored on the backend as encoded fragments. The identifiers of the metadata fragments can be associated with the data object and stored on a low latency frontend storage device. Thus, the amount of metadata per data object stored on expensive low latency frontend storage is reduced to the fragment identifiers. The fragment identifiers can be quickly retrieved, and used to retrieve the identifiers and metadata corresponding to the encoded data fragments from the backend, for retrieval of the data object itself.

Reduced storage of metadata in a distributed encoded storage system

A data object can be encoded into multiple encoded data fragments and stored on the backend of a distributed encoded storage system. The identifiers and metadata corresponding to each fragment of the data object can be stored in a first section of a metadata unit, and the initial part of the data object in a second section. The metadata unit is encoded into multiple metadata fragments, which are stored on the backend. The identifiers of the metadata fragments can be associated with the data object and stored on a fast frontend storage device. In response to a request to access the data object, the identifiers are used to retrieve the metadata fragments from the backend, and decode the metadata unit. The initial part of the data object is retrieved from the metadata unit and transmitted to the requesting client application to begin processing the data object.

Fast read operation utilizing reduced storage of metadata in a distributed encoded storage system

Techniques for distributing data in a distributed data storage system using a hierarchy rule that is generated based on a spreading policy and a set of tolerable failures specified by a user in absence of system deployment information are disclosed. The system includes a controller node which receives a request including a spreading policy and a protection level for spreading a first data object. The controller node determines a hierarchy rule corresponding to the spreading policy based on the protection level. The controller node distributes the first data object in the system using the hierarchy rule and the spreading policy. The controller node receives a reconfiguration of system deployment. The controller node distributes a second data object in the system based on providing protection of the protection level to the second data object without affecting protection of the same protection level applied to the first data object.

Hierarchic storage policy for distributed object storage systems

A set of encoded data fragments is grouped into a container object in sequential order. Each encoded data fragment is a specific fragment size, and the container object is a specific container object size. The sequential order of the set of encoded data fragments can be tracked in a log in memory, such that the location of any one of the data fragments in the container object can be determined. The container object can be stored directly on a specific backend storage element, without using a file system. A corresponding container object identifier identifies the physical storage location of the container object on the backend storage element. The container object identifier is tracked in the log in memory, such that the physical location on the backend storage element of any specific one of the set of encoded data fragments in the container object can be determined.

Growing community of inventors

Huizingen, Belgium

Stijn Blyweert