Sandisk Technologies Inc.

Sandisk Enterprise IP LLC

Wind Jammer Technologies, LLC

Western Digital Technologies, Inc.

Manavalan Krishnan

Brian Walter O'Krafka

Johann George

Ramesh Chander

Niranjan Patre Neelakanta

Darpan Dinker

Tomy Ammuthan Cheru

John Richard Busch

Evgeniy Firsov

Allen Samuels

Frederic H Tudor

Kai Rothauge

Vishal Kanaujia

Vladislav Bolkhovitin

Harihara Kadayam

Nagi Reddy Chodem

Thibash Rajamani

Ajit Kumar Tapse

Executing a query in a disaggregated cluster. A query is received at the disaggregated cluster. A query graph is created based on the query that identifies a hierarchy of vertices, where each vertex is associated with a set of data responsive to at least a portion of the query. The compute nodes process the query graph by first identifying a minimum set of tables, files, and objects stored on the storage nodes required to retrieve data that satisfy the query. Next, the compute nodes selectively assign the identified tables, files, and objects to a leaf vertex of the query graph to optimize retrieving data from the storage nodes. Thereafter, the compute nodes process the retrieved data sets associated with each vertex using separate threads of execution for each vertex of the query graph such that leaf vertices are performed in parallel. The compute nodes then provide a result set.

Disaggregated query processing utilizing precise, parallel, asynchronous shared storage repository access

Systems, methods and/or devices are used to perform memory-efficient mapping of block/object addresses. In one aspect, a method of managing a storage system having one or more storage devices includes a tiered data structure in which each node has a logical ID and entries in the nodes reference other nodes in the tiered data structure using the logical IDs. As a result, when a child node is updated and stored to a new location, but retains its logical ID, its parent node does not need to be updated, because the logical ID in the entry referencing the child node remains unchanged. Further, the storage system uses a secondary mapping table to translate the logical IDs to the corresponding physical locations of the corresponding nodes. Additionally, the secondary mapping table is cached in volatile memory, and as a result, the physical location of a required node is determined without accessing non-volatile memory.

Memory-efficient block/object address mapping

Executing a query in a disaggregated cluster. A query is received at the disaggregated cluster. A query graph is created based on the query that identifies a hierarchy of vertices, where each vertex is associated with a set of data responsive to at least a portion of the query. The compute nodes process the query graph by first identifying all tables, files, and objects stored on the storage nodes whose access is required to retrieve data that satisfy the query. Next, the compute nodes selectively assign the identified tables, files, and objects to a leaf vertex of the query graph to optimize retrieving data from the storage nodes. Thereafter, the compute nodes process the retrieved data sets associated with each vertex using separate threads of execution for each vertex of the query graph such that leaf vertices are performed in parallel. The compute nodes then provide a result set.

Disaggregated query processing on data lakes based on pipelined, massively parallel, distributed native query execution on compute clusters utilizing precise, parallel, asynchronous shared storage repository access

Apparatuses, systems, methods, and computer program products are disclosed for key-value stores with partial data access. An interface module is configured to receive a data object for storage in a key-value store. The data object may include a key and a value. A block object module is configured to generate a plurality of block objects smaller than the data object. A block object may include a new key and a new value. The new key may be based on the key for the data object and on metadata for the new value. The new value may be based on at least a portion of the value for the data object. A storage module is configured to store the block objects in the key-value store.

Key-value store with partial data access

Memory-efficient object address mapping in a tiered data structure

Systems, methods, and/or devices are used to store metadata in a storage system. In one aspect, a first user space module sends a logical memory request to a memory management module of a kernel space module. The logical memory request includes data and metadata. A second user space module obtains the metadata of the logical memory request. A storage engine of the second user space module determines, in accordance with the obtained metadata, a location in non-volatile memory for the data. A second user space module generates a physical memory request including an indication of the non-volatile memory for the data. The second user space module transmits the physical memory request to the kernel space memory management module.

User space data storage management

An apparatus includes a non-volatile memory and a controller coupled to the non-volatile memory. The controller is configured to receive a command from a device to perform a write operation at the non-volatile memory. The command indicates a plurality of logical addresses, data associated with the plurality of logical addresses, and a number of write operations associated with the command.

Group write operations for a data storage device

A method includes accessing, in response to initiating an operation targeting data, auxiliary mapping data to determine whether the auxiliary mapping data includes an indication of a key associated with a node of a hierarchical data structure that is associated with the data. In response to the auxiliary mapping data including the indication of the key, the data is accessed from a memory using a node identification associated with the key. In response to the auxiliary mapping data not including the indication of the key, the data is accessed from the memory using a search operation.

Method and device to access auxiliary mapping data for a data structure

Systems, methods and/or devices are used to coalesce metadata and data writes via write serialization with device-level address remapping. In one aspect, a method of managing a storage system having one or more storage devices includes a serialized write operation to the storage system, in which a serialization segment accumulates data objects and mapping information until the segment is full, at which time the serialization segment is written to the storage system in a single contiguous write. As a result, the number of I/O operations is decreased from a minimum of two (one to write data and one to write updated mapping information) to a single write operation. Further, if the serialization segment contains existing valid data prior to accumulating data objects and mapping information, the valid data is moved to the beginning of the serialization segment using either a remap or xcopy operation.

Coalescing metadata and data writes via write serialization with device-level address remapping

A method of managing a storage system having one or more storage devices includes receiving a request to access, in a tiered data structure, a data object and one or more attributes of the data object and processing the access request. The request includes a first key value for the data object and respective second key values for the one or more attributes of the data object. Modified key values for the attributes of the data object are generated in accordance with at least a portion of the first key value and used to access tree entries for the data object attributes in one or more leaf nodes of the tiered data structure. A tree entry for the data object is also accessed from the same set of one or more leaf nodes.

Reduction of write amplification in object store

An apparatus includes a data sequencing engine configured to receive first data and an identification of a first data container associated with the first data and to allocate the first data to a first data stream of multiple data streams based on the identification. The apparatus further includes at least one memory configured to store the first data container. The data sequencing engine is further configured to cause the at least one memory to send data in the first data stream to a flash memory in response to detecting a data size of the first data stream satisfies a threshold.

Apparatus and method of data sequencing

A memory controller detects a request, with a key to a data object and a value, to perform an operation. The memory controller locates the data object by mapping the key to a location of the data object in a tiered data structure and identifies a sequence number of the data object. When the sequence number of the data object is greater than a sequence number of a snapshot of the tiered data structure, the memory controller replaces the data object with a modified data object, including the key and the value, and assigns the modified object a unique sequence number. When the sequence number of the data object is less than the sequence number of the snapshot, the memory controller inserts a new data object, including the key and the value, into the tiered data structure and assigns the new data object the unique sequence number.

Methods and systems for performing efficient snapshots in tiered data structures

A computer system detects a request to access a first data object stored in a tiered data structure, that includes internal nodes and leaf nodes, where data objects in the leaf nodes include unique key information and corresponding values, and the first data object is uniquely identified by a first key. In response to detecting the request to access the first data object, the computer system retrieves a leaf node that includes the first data object and identifies the first data object in the leaf node by combining unique key information of the first data object with a key prefix that is stored separately in the leaf node to generate a combined key and determining that the combined key matches the first key that uniquely identifies the first data object. After identifying the first data object, the computer system provides access to the first data object.

Compaction of information in tiered data structure

A computer system detects a request to access a first data object stored in a tiered data structure that includes internal nodes and leaf nodes. In response to detecting the request, the computer system determines whether the first data object is stored in a cache that includes a plurality of data objects from the tiered data structure that are stored separately from their corresponding leaf node in the cache. When the first data object is stored in the cache, the computer system returns the first data object from the cache and when the first data object is not stored in the cache, the computer system traverses the tiered data structure to a leaf node that includes the first data object and returns the first data object from the leaf node for the first data object in the tiered data structure.

Efficient cache utilization in a tiered data structure

Approaches for recovering a node of a distributed transactional system. When a recovering node is being brought on-line, a copy of at least a portion of a first data store maintained by an existing node is created without ceasing to process transactions against the first data store at the existing node. The recovering node creates a second data store based on the copy of the first data store. While the recovering node creates the second data store, the recovering node stores committed transaction data received from a plurality of nodes. The committed transaction data describes transactions committed against data stores maintained by the sender of the committed transaction data. The recovering node may thereafter replay, against the second data store, any transactions identified by the committed transaction data that are associated with a global transaction number that is more recent than is a reference commit number.

Growing community of inventors

Fremont, CA, United States of America

Manavalan Krishnan