Microsoft Technology Licensing, LLC

Alan D Halverson

Nikhil Teletia

Srinath Shankar

Jeffrey F Naughton

Rathijit Sen

Willis Lang

Karthik Saligrama Ramachandra

Eric R Robinson

Krishnaram Kenthapadi

Sreenivas Gollapudi

Ajay Kalhan

Srikumar Rangarajan

Jignesh Patel

Sandeep Lingam

Sarvesh Singh

Artem D Yegorov

Vinitha Reddy Gankidi

Prashant Varanasi

Cesar Alejandro Galindo-Legaria

Conor J Cunningham

Hideaki Kimura

Milind Madhukar Joshi

Nina Mishra

Rimma V Nehme

Umar Syed

Kwanghyun Park

Aleksandrs Slivkins

José A Blakeley

Jose Aguilar Saborit

Zhiwei Fan

Kameswara Venkatesh Emani

Harshada Chavan

Methods for data visibility in nested transactions in distributed systems are performed by systems and devices. Distributed executions of queries are performed in processing systems according to isolation level protocols with unique nested transaction identifiers for data management and versioning across one or more data sets, one or more compute pools, etc., within a logical server via a single transaction manager that oversees the isolation semantics and data versioning. A distributed query processor of the systems and devices performs nested transaction versioning for distributed tasks by generating nested transaction identifiers, encoded in data rows, which are used to enforce correct data visibility. Data visibility is restricted to previously committed data from distributed transactions and tasks, and is blocked for distributed transactions and tasks that run concurrently. Local commits for completed transactions and tasks are used to minimize transaction manager interactions, and instant rollbacks are enabled for aborted transactions and tasks.

Data visibility for nested transactions in distributed systems

Distributed query processing is often performed by a set of nodes that apply MapReduce to a data set and materialize partial results to storage, which are then aggregated to produce the query result. However, this architecture requires a preconfigured set of database nodes; can only fulfill queries that utilize MapReduce processing; and may be slowed down by materializing partial results to storage. Instead, distributed query processing can be achieved by choosing a node for various portions of the query, and generating customized code for the node that only performs the query portion that is allocated to the node. The node executes the code to perform the query portion, and rather than materializing partial results to storage, streams intermediate query results to a next selected node in the distributed query. Nodes selection may be involve matching the details of the query portion with the characteristics and capabilities of the available nodes.

Per-node custom code engine for distributed query processing

Methods for automatic selection of degrees of parallelism for efficient execution of queries in a database system are performed by systems and devices. An incoming query associated with a query system is received and features of the incoming query are determined. A system state of the query system and a set of executing queries are also determined, along with a query state of each executing query in the set. At runtime of the incoming query, allocation of a degree of parallelism for executing the query is determined by calculating different possible execution times for the incoming query at least partially concurrently with the set of executing queries. Execution times are calculated for different parallel thread options and based on query features, system state, or query states of executing queries. The execution of the incoming query is initialized with the parallel thread option corresponding to a specific execution completion time.

Methods for automatic selection of degrees of parallelism for efficient execution of queries in a database system

A split-index can be employed for access to external data. The index can be created on a primary data storage system for data stored externally on a secondary data storage system. After creation, the index can be utilized to expedite at least query execution over the externally stored data. The index can be updated upon detection of changes to data. Further, even when the index is not completely up to date, the index can be exploited for query execution. Furthermore, hybrid execution is enabled with the index and without the index.

External data access with split index

According to examples, an apparatus may include a machine readable medium on which is stored machine readable instructions that may cause a processor to, for each of a plurality of resource setting levels, determine resource usage characteristics and execution times of executed workloads, assign, based on the resource usage characteristics of the executed workloads, each of the executed workloads into one of a plurality of resource bins, determine, for each of the resource bins, an average execution time of the executed workloads in the resource bin, determine a total average execution time of the executed workloads from the determined average execution times, identify a lowest total average execution time of the determined total average execution times, determine the resource setting level corresponding to the identified lowest total average execution time, and tune a resource setting to the determined resource setting level.

Tune resource setting levels for query execution

A frequency governing method for a processor includes reading power management information from the processor. The processor operates in an active state or one of multiple power saving states. The power management information includes first information indicating occupancy of a first core of the processor in the active state and second information indicating occupancy of the first core in a first power saving state. The method includes generating an effective utilization based on the first information and the second information. The method includes setting a target frequency by performing a selected action of a first action including increasing the target frequency and a second action including decreasing the target frequency with respect to a current frequency of the processor based on the effective utilization. The selected action is selected according to the effective utilization. The method includes instructing the processor to operate the first core at the target frequency.

Core frequency management using effective utilization for power-efficient performance

Processing a database query. A method includes receiving a database query from a user. The database query includes one or more imperative functions. The one or more imperative functions are converted to one or more declarative query representations. The one or more declarative query representations include standardized relational operators included in a relational query language. Further, the one or more declarative query representations are optimizable by a query optimizer of the database. The database query is optimized at the query optimizer to create a query plan by evaluating any declarative query representation originally in the database query received from the user and the one or more declarative query representations.

Method for optimization of imperative code executing inside a relational database engine

The present invention extends to methods, systems, and computer program products for formulating global statistics for parallel databases. In general, embodiments of the invention merge (combine) information in multiple compute node level histograms to create a global histogram for a table that is distributed across a number of compute nodes. Merging can include aligning histogram step boundaries across the compute node histograms. Merging can include aggregating histogram step-level information, such as, for example, equality rows and average range rows (or alternately equality rows, range rows, and distinct range rows), across the compute node histograms into a single global step. Merging can account for distinct values that do not appear at one or more compute nodes as well as distinct values that are counted at multiple compute nodes. A resulting global histogram can be coalesced to reduce the step count.

Optimizing queries using global statistics for distributed databases

Defragmenting a cluster service to service additional capacity requests on the service. A method includes determining an amount of server resources needed for an additional deployment reservation request for a new deployment or increasing reservation of resources of an existing deployment. The method further includes determining a server that currently does not have capacity to service the additional deployment reservation request. The method further includes determining how resources on the server can be freed up by moving other replicas of other deployments on the server to other servers to allow the server to service the additional deployment reservation request.

Defragmenting clusters with reserved resources

The present invention extends to methods, systems, and computer program products for executed stored procedures at parallel databases. Stored procedures are transformed so that execution of the stored procedure is split between a standalone database server and a parallel database coordinator. Execution of the stored procedure is initiated at the standalone database server. At execution time, control-flow statements, variable assignment, expression evaluation, etc., are handled by the standalone database server. SQL statements are passed from the standalone database server to the database for the execution. Results from executed SQL statements can be returned to the standalone database server or to a client. The parallel database coordinator can be added as a linked server to the standalone database server. In some embodiments, a session token is used to share session state between different parties.

Executing stored procedures at parallel databases

Distributing rows of data in a distributed table distributed across a plurality of nodes. A method includes identifying skewed rows of a first table to be distributed in a distributed database system. The skewed rows include a common data value in a column such that the skewed rows are skewed, according to a predetermined skew factor, with respect to other rows in the first table not having the common data value. Non-skewed rows of the first table that are not skewed according to the skew factor are identified. The skewed rows of the first table are distributed across nodes in a non-deterministic fashion. The non-skewed rows of the first table are distributed across nodes in a deterministic fashion. The rows of the first table distributed across the nodes, whether distributed in a deterministic fashion or non-deterministic fashion, are stored in a single table at each of the nodes.

Skew-aware storage and query execution on distributed database systems

A single system image is provided for a parallel data warehouse system by exposing a shell database within a database management system comprising metadata and statistics regarding externally stored data. Further, functionality of the database management system can be exploited to perform pre-execution tasks. In one instance, one or more execution plans can be generated by the database management system for an input command and subsequently employed to generate a distributed execution plan.

Growing community of inventors

Sunnyvale, CA, United States of America

Alan D Halverson