Microsoft Technology Licensing, LLC

Nikhil Teletia

Alan D Halverson

Jignesh Patel

Kwanghyun Park

Jae Young Do

Willis Lang

Ajay Kalhan

Srinath Shankar

Artem D Yegorov

Vinitha Reddy Gankidi

Prashant Varanasi

Jeffrey F Naughton

Hideaki Kimura

Milind Madhukar Joshi

Karthik Saligrama Ramachandra

José A Blakeley

Jose Aguilar Saborit

Alan Dale Halverson

Distributed query processing is often performed by a set of nodes that apply MapReduce to a data set and materialize partial results to storage, which are then aggregated to produce the query result. However, this architecture requires a preconfigured set of database nodes; can only fulfill queries that utilize MapReduce processing; and may be slowed down by materializing partial results to storage. Instead, distributed query processing can be achieved by choosing a node for various portions of the query, and generating customized code for the node that only performs the query portion that is allocated to the node. The node executes the code to perform the query portion, and rather than materializing partial results to storage, streams intermediate query results to a next selected node in the distributed query. Nodes selection may be involve matching the details of the query portion with the characteristics and capabilities of the available nodes.

Per-node custom code engine for distributed query processing

A split-index can be employed for access to external data. The index can be created on a primary data storage system for data stored externally on a secondary data storage system. After creation, the index can be utilized to expedite at least query execution over the externally stored data. The index can be updated upon detection of changes to data. Further, even when the index is not completely up to date, the index can be exploited for query execution. Furthermore, hybrid execution is enabled with the index and without the index.

External data access with split index

Replicating a primary application cache that serves a primary application on one network node into a secondary application cache that serves a secondary application on a second network node. Cache portions that are within the primary application cache are identified, and then identifiers (but not the cache portions) are transferred to the second network node. Once these identifiers are received, the cache portions that they identify may then be retrieved into the secondary application caches. This process may be repeatedly performed such that the secondary application cache moves towards the same state as the primary application cache though the state of the primary application cache also changes as the primary application operates by receiving read and write requests.

Application cache replication to secondary application(s)

Data-manipulation-language statements can target data in data store of a data storage system. Data manipulations specified in a statement, including in-place manipulations, can be recorded and maintained. Application of data manipulations to the store can be avoided or potentially initiated at a later time. Prior to application of the data manipulations to the data store, queries over data in the data store can be processed in view of the recorded data manipulations and data acquired from the data store, such that the data manipulations are reflected in the data acquired from the data store.

Efficient data manipulation support

Replicating a primary application cache within a secondary application cache

The automated recovery of the warmth of cache of an application that has been subject to a running state change that degraded the warmth of the cache. To prepare for a loss in warmth, the cache portion identifiers are captured, and corresponding cache portions are stored in an external store. Thereafter, the application experiences changes in state. In order to warm the application cache more quickly, cache portions identified by the captured cache portion identifiers are retrieved from the external data store and placed in the application cache.

Automatic recovery of application cache warmth

Defragmenting a cluster service to service additional capacity requests on the service. A method includes determining an amount of server resources needed for an additional deployment reservation request for a new deployment or increasing reservation of resources of an existing deployment. The method further includes determining a server that currently does not have capacity to service the additional deployment reservation request. The method further includes determining how resources on the server can be freed up by moving other replicas of other deployments on the server to other servers to allow the server to service the additional deployment reservation request.

Defragmenting clusters with reserved resources

The present invention extends to methods, systems, and computer program products for executed stored procedures at parallel databases. Stored procedures are transformed so that execution of the stored procedure is split between a standalone database server and a parallel database coordinator. Execution of the stored procedure is initiated at the standalone database server. At execution time, control-flow statements, variable assignment, expression evaluation, etc., are handled by the standalone database server. SQL statements are passed from the standalone database server to the database for the execution. Results from executed SQL statements can be returned to the standalone database server or to a client. The parallel database coordinator can be added as a linked server to the standalone database server. In some embodiments, a session token is used to share session state between different parties.

Executing stored procedures at parallel databases

The automated recovery of the warmth of cache of an application that has been subject to a running state change that degraded the warmth of the cache. To prepare for a loss in warmth, the state of a cache portion identifiers are captured. Such identifies the cache portions that are within the application cache at an instant in time. Thereafter, the application experiences a change in running state that diminishes the warmth of the application cache. For instance, the application might be stopped and restarted. After and despite this cache degradation, the application may continue to operate. However, in order to warm the application cache more quickly, while the application is operating, the application cache is automatically warmed. For instance, while the application is operating, cache portions identified by the captured cache portion identifiers are automatically accessed from a source that is external to the application cache.

Distributing rows of data in a distributed table distributed across a plurality of nodes. A method includes identifying skewed rows of a first table to be distributed in a distributed database system. The skewed rows include a common data value in a column such that the skewed rows are skewed, according to a predetermined skew factor, with respect to other rows in the first table not having the common data value. Non-skewed rows of the first table that are not skewed according to the skew factor are identified. The skewed rows of the first table are distributed across nodes in a non-deterministic fashion. The non-skewed rows of the first table are distributed across nodes in a deterministic fashion. The rows of the first table distributed across the nodes, whether distributed in a deterministic fashion or non-deterministic fashion, are stored in a single table at each of the nodes.

Skew-aware storage and query execution on distributed database systems

The present invention extends to methods, systems, and computer program products for performing parallel joins on distributed database data. Embodiments of the invention include a phased semi-join reduction strategy using replication and shuffle operations to join a first and a second data source. A filter building phase uses replication and pushes down a 'Distinct' (e.g., SQL) operator to produce a list of join keys for the first data source (one side of the join). A shuffle phase for the second data source is modified to join to the key list produced in the first phase as a row filtering mechanism. A join phase then joins the first and second data sources.

Growing community of inventors

Madison, WI, United States of America

Nikhil Teletia