Cloudera, Inc.

Okera, Inc.

Databricks Inc.

Nong Li

Henry Noel Robinson

Justin Erickson

Marcel Kornacker

Lenni Kuff

Amandeep Khurana

Alan Choi

Alex Behm

Itay Alfred Neeman

Various embodiments of the present technology generally relate to management of big data storage and the physical removal of data via data access systems for large data processing environments having multiple application services and multiple storage services. In some embodiments, a method of physically removing data from a storage system provides for identifying one or more files needing data removal treatment, determining that a file needing data removal treatment should be queued, and populating a queue with the file. Determining that a file should be queued is based, at least in part, on a staleness tolerance. The method further provides for treating the file and replacing a previous version of the file in storage with the updated file. In some implementations, treating the file includes removing data from the file to create an updated file and may further include additional changes to the file.

Background dataset maintenance

A system manages access control policies for accessing data stored in a plurality of data platforms. The system receives access control policy specification describing an access control policy that controls access to a set of datasets by a set of users. The set of datasets is defined based on a condition based on the data tags representing attributes of the datasets. The system compiles the access control policy specification to generate a platform independent access control representation of the access control policy. The platform independent access control representation comprises a set of tuples. Each tuple identifies a particular set of users, a particular set of datasets, and a particular action. The system further generates data platform specific instructions for each data platform of the plurality of data platforms.

Synchronization of access control policies with external data platforms

Disclosed herein provides enhancements for operating a data access system for large data processing environments. In one implementation, a method provides for maintaining a data structure comprising a plurality of customized code configurations each associated with a data request rule for each of the multiple application services. A code configuration query from a user is then received indicating a data request rule. The code configuration query requests code configurations for data retrieval from at least one of the multiple storage services over the data access system. The data structure is queried for one or more customized code configurations for each of the multiple application services associated with the indicated data request rule. The user is then provided with the one or more customized code configurations for each of the multiple application services associated with the indicated data request rule.

Customized code configurations for a multiple application service environment

Disclosed herein provides enhancements for operating a data access application service executing on a data access server system and an external computing system. In the data access server system, a request is received from a client device executing at least one of multiple application services for a dataset from one or more of multiple storage systems. In the data access server system, a data retrieval instruction is generated for the client device to access the dataset from the one or more of the multiple storage systems. The data retrieval instruction comprises task descriptions and a temporary credential. The data retrieval instruction is transferred to the external computing system via the client device and the requested dataset is retrieved and deployed based on the task descriptions and the temporary credential from the one or more of the multiple storage systems.

Data retrieval using distributed workers in a large-scale data access system

Various embodiments of the present technology generally relate to management of big data storage and data access control systems. In some embodiments, a data access system for use in multiple application service and multiple storage service environments comprises a sandbox database for users, wherein the sandbox database is a virtual database environment via which a user may access datasets according to one or more access policies. In some embodiments, the data access system receives a user request to access a dataset stored in a database into the sandbox environment, wherein the database is associated with the data access system. In response to the request, the data access system may retrieve the corresponding data from the database, determine any associated sandbox access policies, and generate an anonymized data table in the sandbox environment.

Systems and methods for a virtual sandbox database

A format conversion engine for Apache Hadoop that converts data from its original format to a database-like format at certain time points for use by a low latency (LL) query engine. The format conversion engine comprises a daemon that is installed on each data node in a Hadoop cluster. The daemon comprises a scheduler and a converter. The scheduler determines when to perform the format conversion and notifies the converter when the time comes. The converter converts data on the data node from its original format to a database-like format for use by the low latency (LL) query engine.

Background format optimization for enhanced queries in a distributed computing cluster

Disclosed herein provides enhancements for operating a data access system for large data processing environments. In one implementation, a method provides for receiving a data query from at least one of the multiple application services and identifying metadata that defines policies for deploying the queried data. The method further provides retrieving the queried data from at least one of the multiple storage services, generating a data configuration containing the retrieved data based on standardized parameters and the policies defined by the metadata, and deploying the data configuration to the at least one of the multiple application services.

Generation of data configurations for a multiple application service and multiple storage service environment

Background format optimization for enhanced SQL-like queries in Hadoop

A low latency query engine for APACHE HADOOP&#x2122; that provides real-time or near real-time, ad hoc query capability, while completing batch-processing of MapReduce. In one embodiment, the low latency query engine comprises a daemon that is installed on data nodes in a HADOOP&#x2122; cluster for handling query requests and all internal requests related to query execution. In a further embodiment, the low latency query engine comprises a daemon for providing name service and metadata distribution. The low latency query engine receives a query request via client, turns the request into collections of plan fragments and coordinates parallel and optimized execution of the plan fragments on remote daemons to generate results at a much faster speed than existing batch-oriented processing frameworks.

Growing community of inventors

San Francisco, CA, United States of America

Nong Li