International Business Machines Corporation

Amazon Technologies, Inc.

Google Inc.

Servicenow, Inc.

Yuri L Volobuev

Frank Bernhard Schmuck

Demyn Lee Plantenberg

Murali Brahmadesam

Anoop Kochummen Johnson

Thibaud Baptiste Hottelier

Justin Levandoski

David Jones Craft

Alessandro Sorniotti

James Christopher Wyllie

Felipe Knop

Raman Mittal

Anil Kurmus

Punit Rajgaria

Kuei-Yu Wang-Knop

Zhiyong Tian

Shriram Sridharan

Li-Che Hsiao

Brian P Dixon

Deepak Choudhary Nettem

Stefano Stefani

Anurag Windlass Gupta

Gopi K Attaluri

Saileshwar Krishnamurthy

Alexandre Olegovich Verbitski

Kamal Kant Gupta

Tengiz Kharatishvili

Jaeyun Noh

Mingge Deng

Jeffrey Davis

Debanjan Saha

Aakash Shah

Bigang Li

Shuang Guan

Sangil Song

Nanda Kaushik

Dhruva Ranjan Chakrabarit

Aleksandr Valerevich Feinberg

Namrata Bapat

Gaurav Saxena

Victor Sergeyevich Agababov

Sungheun Wi

Volodymyr Verovkin

A method for unstructured data analytics in data warehouses includes receiving an unstructured data query from a user, the unstructured data query requesting the data processing hardware determine one or more unstructured data files stored at a data repository that match query parameters. The method includes determining, using an object table, a set of unstructured data files stored at the data repository that matches the query parameters. The object table includes a plurality of rows, each row of the plurality of rows associated with a respective unstructured data file stored at the data repository, and a plurality of columns, each column of the plurality of columns comprising metadata associated with the respective unstructured data file of each row of the plurality of rows. The method includes returning, to the user, a structured data table including the determined set of unstructured data files.

Unstructured data analytics in traditional data warehouses

Aspects of the disclosure are directed to merging data lake openness with scalable metadata for managed tables in a cloud database platform, allowing for atomicity, consistency, isolation, and durability (ACID) transactions, performant data manipulation language (DML), higher throughput stream ingestion, data consistency, schema evolution, time travel, clustering, fine-grained security, and/or automatic storage optimization. Table data is stored in various open-source file formats in cloud storage while physical metadata of the table data is stored in a scalable metadata storage system.

Managed tables for data lakes

Remote grouping operations may be used to perform queries for distinct values. A query may be received at a query engine that includes a request for distinct items from a data set stored in a data store. To perform the query, a request may be made to the data store to group items from the data set by value when returned from the data store to the query engine. The query engine may then evaluate the items grouped by value that are returned from the data store to determine distinct items as part of performing the query.

Remote grouping operations to perform queries for distinct values

Aspects of the disclosure are directed to a metadata cache for extending data warehouse features to data lakes. The metadata cache can accelerate query execution by directly accessing unmanaged data from the data lake rather than accessing the data through the data warehouse. The metadata cache can allow for filtering the unmanaged data to improve the speed of retrieving data for executing a query.

Query acceleration and metadata caching

Direct storage loading may be used to add data to a database. New data may be added to a database, using nodes different than a database engine to access a database. The addition of the new data may be assigned to different nodes. The nodes may obtain the data and store the data to storage locations according allocated space in the database by the database engine. The new data can then be made available for access at the database engine.

Direct storage loading for adding data to a database

Storage level query processing may be implemented for processing database queries. Nodes that can access a database may perform parallel processing for at least a portion of a database query. An indication may be received that specifies parallel processing for the database query. The nodes can then be caused to perform the portion of the query as part of providing a result in response to the database query instead of a node, such as a query engine node, that received the database query.

Storage level parallel query processing

An identification system that may be used in heterogeneous computing environments provides a fail-free path to providing identifiers from a single canonical namespace. Objects or gateways requiring an identifier for access are accessed using an identifier for the canonical namespace. If an entity requests access using an identifier from another namespace, an external database is consulted to determine if a mapping exists for the identifier to another identifier the canonical namespace. If no mapping exists, or the external database is unavailable, then an identifier is automatically generated in the canonical namespace and is used for the access. An internal database is updated with the automatically generated identifier, providing a mechanism to add mappings without administrative intervention. To access resources requiring an identifier from another particular namespace, a canonical namespace identifier may be mapped to another identifier in the particular namespace, or a generic identifier may be used.

Unified user identification with automatic mapping and database absence handling

A computer-implemented method of encryption of several units of a computerized system, wherein each of the units comprises data, includes generating distinct initialization vectors, or IVs, for the units, and storing the generated IVs; and for each unit of the several units: accessing a stored IV corresponding to the unit; and encrypting the unit according to the accessed IV and an encryption key.

Initialization vectors generation from encryption/decryption

Embodiments relate to dynamically allocating access to a shared resource based on quota shares. An aspect includes computing, by a client node, an expected size of quota shares needed by the client node based on a historical rate of resource consumption by the client node. The client node transmits a request for a count of quota shares based on the expected size. The client node receives an indication that the request for the count of quota shares is one of: granted, denied, and granted-in-part. The client node self-rules the shared resource based on the indication.

Dynamic and adaptive quota shares

The disclosed technology identifies bottlenecks in a hierarchical storage subsystem and, based upon the rate at which data may be transmitted through a particular bottleneck, determines the smallest number of disk drives required to match that transmission rate. If the required number of disks is less than the total number of disks, only a subset of the total number are maintained in an active state with the remainder places in either a 'standby' or 'off' mode. In this manner, overall system power consumption is reduced. In one embodiment, the disclosed techniques are implemented by active disk management at high level of storage infrastructure.

Advanced disk drive power management based on maximum system throughput

Growing community of inventors

Austin, TX, United States of America

Yuri L Volobuev