Oracle International Corporation

Qubole, Inc.

Microsoft Technology Licensing, LLC

Shrikanth Shankar

Ananth Raghavan

Subhransu Basu

Vikram Shukla

Ramesh Kumar

Badhri G Varanasi

Thierry Cruanes

William H Waddington

Kumar Rajamani

Bhaskar Ghosh

Jaebock Lee

Joydeep Sen Sarma

Mayank Ahuja

Russell John Green

Benoit Dageville

Amit Ganesh

Juan R Loaiza

Jagannathan Srinivasan

Dheeraj Pandey

Souripriya Das

George Eadon

Vasudha Krishnaswamy

Eugene Inseok Chong

Sivaramakrishnan Narayanan

Krishna Kunchithapadam

Khai Tran

Chandrasekharan Iyer

Maneesh Varshney

Philip Yam

David M Alpern

Rushan Chen

Walaa Eldin M Moustafa

Joy Forsythe

Amirali Valiani

Guhan Viswanathan

Ho Chak Hung

Adwait N Tumbde

Alan Choi

Ratandeep S Ratti

Carl W Steinbach

Vasanth Rajamani

Suja Viswesan

Systems and methods for automatically scaling a big data system. Methods include determining, at a first time, a first number of nodes for a cluster to process a request; assigning an amount of nodes equal to the first number of nodes to the cluster; determining a rate of progress of the request; determining, at a second time based on the rate of progress a second number of nodes; and modifying the amount of nodes to equal the second number of nodes. Systems include a cluster manager, to add and/or remove any nodes; the big data system, to process requests that utilize the cluster and nodes, and an automatic scaling cluster manager including a big data interface for communicating with the big data system; a cluster manager interface for communicating with the cluster manager; and a cluster state machine.

Systems and methods for auto-scaling a big data system

The disclosed embodiments provide a system for processing data. During operation, the system obtains a user-defined function (UDF) that is defined using a standard UDF interface for accessing multiple platforms. Next, the system converts data in a platform into a standard form that is supported by the standard UDF interface using a platform-specific wrapper for the UDF and a platform-specific implementation of the standard UDF interface. The system then generates output data in the standard form by applying one or more operations to the data in the standard form. Finally, the system converts the output data into a platform-specific form that is supported by the platform and returns the output data in the platform-specific form.

Platform-independent user-defined functions for data access

The present invention is generally directed to a distributed computing system comprising a plurality of computational clusters, each computational cluster comprising a plurality of compute optimized instances, each instance comprising local instance data storage and in communication with reserved disk storage, wherein processing hierarchy provides priority to local instance data storage before providing priority to reserved disk storage.

High performance hadoop with new generation instances

When altering records in a repository of information to add an attribute that has a non-NULL default value and a corresponding constraint to the records, no update record is issued and instead, the existing records are essentially treated as having a NULL value for the attribute being added. To compute a query over the records to which the attribute has been added, a value is generated for rows that hold NULL values for the attribute in which the value in which the NULL values held therein are translated to the default value.

Transforming default values dynamically

A database may facilitate zero-downtime upgrades by concurrently maintaining multiple editions of database objects for use by both pre-upgrade and post-upgrade clients of a database application. Operations performed within the database are associated with an edition based on, for example, an initiating client or transaction. When an operation references an object or data, the database automatically performs the operation using the object or data associated with the edition with which the operation is itself associated. The database may determine the associated edition without explicit identification of the associated edition in a query or in code. Thus, no client or stored procedure code changes are necessary to reflect a new edition added during an update. Data changes in one edition may be automatically and immediately propagated to the other edition through the use of cross-edition triggers, thereby allowing both pre-upgrade and post-upgrade clients to remain fully functional throughout an upgrade.

Low-downtime and zero-downtime upgrades of database-centric applications

A method and apparatus for referencing a partition of a data storage space in a request to perform an operation with relation to the partition are disclosed. The techniques disclosed do not rely on knowing the partition's name, thus simplifying the process of requesting a database to perform an operation on a partition when that partition's name is unknown. Data such as a DDL statement is received. The data indicates a value in a FOR clause in lieu of the partition name. The data also indicates an operation to be performed with respect to a particular partition of the partitioned space. The particular partition is not named in the received data. Based on the value, it is determined that the particular partition is a first partition. The operation is performed with respect to the first partition.

Referring to partitions with for (values) clause

A method and apparatus for managing data in a partitioned data storage space is provided. Data referencing a new data item to be stored in the data storage space is received. It is determined that the data item does not pertain to any previously created partition. In response to the determination, a new partition is automatically created in the data storage space. The partition is mapped to a new set of values, such as to a next unpartitioned interval of dates or numbers.

Dynamic creation of database partitions

Techniques are provided for processing a CREATE INDEX statement that avoids one or more locks on the underlying table. The processing of such a statement may be divided into a build phase and a cooperation phase. During the build phase, a journal table is created and published. A snapshot of the table is obtained after the last DML transaction (that was pending at the time the journal table was published) commits. The index is built based on the state of the table as of the snapshot. Any changes to the table while the index is built are recorded in the journal table. During the cooperation phase, the index is partially published. A subsequent DML statement determines whether any changes in the journal table affect the same data as the subsequent DML statement. If so, then those changes are applied to the index followed by the subsequent DML statement.

Online index builds and rebuilds without blocking locks

A method and apparatus for changing versioned metadata without invalidating cursors is provided. Cursors may be linked to the appropriate version of metadata at execution time as opposed to compilation time, meaning that they do not require recompilation to link to a new version of metadata. A database system maintains compiled code implementing a plan with respect to data items in a data storage space. The compiled code may have been generated in response to receiving a request to perform an operation that may be implemented using the plan. A version of metadata is also maintained. The metadata describes characteristics of the data storage space implicated by the plan. The compiled code is executed. The step of execution comprises binding the compiled code to the version of metadata. A proxy object referenced in the compiled code may facilitate the binding of the compiled code to the appropriate version of metadata.

Changing metadata without invalidating cursors

A method and apparatus for the on-demand allocation of segments and creation of metadata for previously-created data storage spaces and partitions are provided. A space is created in a database. As part of this creation process, no segment is allocated for the space. Rather, metadata describing the space sufficiently to allocate the segment in the future is created and maintained by the database. Data is received indicating a new item. Based on the metadata, it is determined that the new item pertains to the space. In response to the determination, a segment is allocated for the space. Additional metadata necessary for normal database operations in relation to the space and newly-allocated segment may also be created at this time. The new item can then be stored in the space.

Partition/table allocation on demand

Techniques for a data definition language (DDL) statement acquiring a lock on a table in wait mode in a deadlock free manner are provided. A DDL statement specifies metadata of a table that is to be modified, whereas a data manipulation language (DML) statement specifies a table to be modified. When a DDL statement is received, a proxy statement is initiated and executed instead of first acquiring an exclusive lock on metadata of a particular table. The proxy statement is similar to a DML statement in that the proxy statement acquires a lock on the particular table. After the proxy statement has been processed, an exclusive lock on the metadata is acquired.

Acquiring locks in wait mode in a deadlock free manner

Versioned metadata allows multiple versions of metadata for a given object to exist concurrently on a system thereby supporting the parallel execution of operations which would otherwise be mutually exclusive. Metadata updates are permitted while other objects maintain access to another version of the metadata through a versioning infrastructure. The versioning infrastructure allows the creation of new versions of metadata, maintains obsolete versions in the system as long as they are being used by any object, and deletes the obsolete versions once they are no longer in use.

System and method for managing versions of metadata

Techniques are described for performing partition pruning in situations where previous database servers could not. Specifically, techniques are provided for (1) performing partition pruning based on predicates that involve non-prefix subsets of composite partitioning keys, (2) performing partition pruning when a global index access path is used to execute a query, and (3) performing partition pruning when predicates on partitioning columns are OR-ed together. More generally, techniques are described for partition pruning based on a non-contiguous set of not-pruned partitions.

Techniques for partition pruning

Techniques are provided for generating execution plans for, and executing, database statements that contain filter subqueries. Upon receiving a database statement that contains a filter subquery, the database server builds an execution plan that includes a filter operation that is fed by a subtree that corresponds to the subquery. The database server performs a cost analysis to determine whether (1) the filter operation should be parallelized, and (2) the subquery tree should be parallelized. Techniques are also described for generating plans and executing queries where (1) both the filter operation and the subquery subtree are parallelized, (2) the filter operation is parallelized but the subquery subtree operation is performed serially, and (3) the filter operation is performed serially but the subquery subtree operation is parallelized.

Executing filter subqueries using a parallel single cursor model

Systems, methodologies, media, and other embodiments associated with supporting reference partitioned tables in a relational database are described. One example method includes identifying a partitioning strategy (e.g., range, list, hash) associated with a parent table that is related to a child table by a referential constraint. The example method may also include creating the child table to be a reference-partitioned table partitioned according to the partitioning strategy associated with the parent table. The example method may also include creating the child table to be equi-partitioned with respect to the parent table. The child table is not to replicate a partition key of the parent table but rather is to inherit a partitioning key associated with the parent table through a foreign key relationship.

Reference partitioned tables

Growing community of inventors

Mountain View, CA, United States of America

Shrikanth Shankar