Microsoft Technology Licensing, LLC

Microsoft Corporation

Ronnie Chaiken

David Mitford Gillies

Andrew James Edwards

William Ramsey

Atul Adya

William J Bolosky

Hoi Huu Vo

Amitabh Srivastava

Ken B Pierce

John A Lefor

Oliver H Foehr

Gideon Andreas Yuval

Jacob Rubin Lorch

Hon Keat W Chan

Jiyang Liu

Bruce M Kuramoto

Gregory A Eigsti

Alastair Wolman

Jitendra Dattatraya Padhye

Sharad Agarwal

Per-Ake Larson

Stefan Saroiu

Ratul Mahajan

Jingren Zhou

David Alexander Molnar

Weidong Cui

Lenin Ravindranath Sivalingam

Marcus Jon Jager

Bryan Jeffrey Parno

Shahin Shayandeh

Dzmitry Suponau

Ian Obermiller

Sirius Kuttiyan

Bikas Saha

James David Ryseff

Christopher M Moore

Christopher Stephen Frederick Smowton

Pre-Ake Larson

S Craig Schertz

A system and method to process a dataset with a database management system (DBMS) engine. The method includes splitting bulk data into a plurality of chunks. The method also includes converting the chunks to a plurality of row groups. The row groups are a dataset external to a DBMS comprising the DBMS engine. The method further includes creating an empty DBMS table within the DBMS. Additionally, the method includes attaching the dataset external to the DBMS to the empty DBMS table. The method also includes executing a MapReduce job on a cluster of compute nodes, using the dataset external to the DBMS as input.

Processing datasets with a DBMS engine

The insertion and processing of programming language code (e.g., C#) in SQL statements, and the dynamic compiling of the code to detect errors prior to statement execution. The SQL statement with arbitrary programming language code can then be executed concurrently as a query on a computer cluster. External libraries can be imported and the associated functionality leveraged from within a SQL statement. The programming language compiler performs checking at build time, rather than retuning an error during the statement execution against the cluster, a potentially costly proposition in both computing time and human resources.

Programming language extensions in structured queries

Methods and systems for dynamically providing application analytic information are provided herein. The method includes inserting instrumentation points into an application file via an application analytic service and dynamically determining desired instrumentation points from which to collect application analytic data. The method also includes receiving, at the application analytic service, the application analytic data corresponding to the desired instrumentation points and analyzing the application analytic data to generate application analytic information. The method further includes sending the application analytic information to a client computing device.

Dynamically providing application analytic information

Described is a technology by which the identity of a person (e.g., a customer in a commercial transaction) is determinable without active identification effort, via biometric data is obtained without action by the person. Machine processing of the biometric data over a set of possible persons, determined from secondary proximity sensing, is used to determine or assist in determining the identity of the person.

Personal identification combining proximity sensing with biometrics

An optimizer uses comprehensive reasoning regarding partitioning, sorting, and grouping properties for query optimization. When optimizing an input query expression, logical exploration generates alternative logical expressions. Physical optimization explores physical operator alternatives for logical operators. Required partitioning, sorting, and grouping properties of inputs to physical operators are determined. Additionally, delivered partitioning, sorting, and grouping properties of outputs from physical operators are determined. In some embodiments, enforcer rules are employed to modify structural property requirements to introduce alternatives for consideration. Property matching identifies valid execution plans in which the delivered partitioning, sorting, and grouping properties satisfy corresponding required partitioning, sorting, and grouping properties. An execution plan having the lowest cost is selected as the optimized execution plan.

Exploiting partitioning, grouping, and sorting in query optimization

Systems, methods, and computer media for scheduling vertices in a distributed data processing network and allocating computing resources on a processing node in a distributed data processing network are provided. Vertices, subparts of a data job including both data and computer code that runs on the data, are assigned by a job manager to a distributed cluster of process nodes for processing. The process nodes run the vertices and transmit computing resource usage information, including memory and processing core usage, back to the job manager. The job manager uses this information to estimate computing resource usage information for other vertices in the data job that are either still running or waiting to be run. Using the estimated computing resource usage information, each process node can run multiple vertices concurrently.

Concurrent data processing in a distributed system

Core commands and aggregations of such commands are provided to programmers to enable them to generate programs that can be parallel-processed without requiring the programmer to be aware of parallel-processing techniques. The core commands and aggregations abstract mechanisms that can be executed in parallel, enabling the programmer to focus on higher-level concepts. The core commands provided include commands for applying a function in parallel and distributing and joining data in parallel. The output of each core command can implement an interface that can enable underlying mechanisms to stitch together multiple core commands in a cohesive manner to perform more complex actions.

Disco: a simplified distributed computing library

Scripting core commands and aggregations of such commands are provided to script authors to enable them to generate scripts that can be parallel-processed without requiring the author to be aware of parallel-processing techniques. The scripting core commands and aggregations abstract mechanisms that can be executed in parallel, enabling the script author to focus on higher-level concepts. The scripting core commands provided include commands for applying a function in parallel and distributing and joining data in parallel. For added flexibility, one or more scripting core commands can utilize functions written in a different programming language and referenced appropriately in code blocks.

Discoscript: a simplified distributed computing scripting language

Structured queries, such as those written using the Structured Query Language (SQL) can be an efficient mechanism for expressing the information sought to be obtained from a collection of data. Mechanisms can enable the use of structured queries to express data processing that can be performed in parallel in order to obtain the attendant efficiencies of such processing. The structured queries, whether standing alone or integrated into another programming context, can be translated into an intermediate form that can be compatible with, or equivalent to, the programming context into which they were integrated or some other high-level programming language. The intermediate, translated form can use core commands that abstract mechanisms that can be executed in parallel. The core commands include commands for applying a function in parallel and distributing and joining data in parallel, and also include aggregations of core commands directed to commonly performed functions.

DISCOSQL: distributed processing of structured queries

Efficient processing of time-bound messages is described herein. In an implementation, messages are classified either time-bounded messages or non-time-bounded messages. The time-bounded messages are then processed separately from the non-time-bound messages. Examples of time-bounded messages can include spam e-mails or other types or classes of e-mails subject to a retention policy, such that the are retained only for some set period of time. Metadata relating to the time-bound messages are compiled and stored separately from the messages, thereby enabling optimization of the metadata storage and processing apart from the messages. The time-bounded messages are accumulated before they are processed in bulk, thereby reducing the number if I/O operations consumed by the time-bound messages, and reducing the amount and cost of resources supporting the process of the time-bounded messages.

Efficient processing of time-bounded messages

Ambiguous stack references in low level software instructions can be resolved when raising a representation of low level software instructions, such as native code. The number of parameters passed to a function can be determined by heuristics, including direct call and indirect call heuristics. Indirect call heuristics can include dataflow-to-target-function, dataflow-to-target-type, import name, and cleanup instructions heuristics. A control flow heuristic can use an alternating direction propagation of seed points to determine the stack height at basic block boundaries. Interpolation can be used within the basic blocks as needed. The raised representation can be used for static analysis of the low level software instructions.

Raising native software code

Described are systems and techniques for losslessly restarting subsystems in a distributed file system. By partitioning functionality and logging appropriately across the kernel and user-level boundaries on a client, the user-level subsystem may be made losslessly restartable. In particular, a map assisted state transfer may include receiving one or more state updates, marshaling one or more active data-structures into a marshaled shadow, applying the received state updates to the marshaled shadow and re-instantiating the active data-structures by unmarshaling the marshaled shadow.

Lossless recovery for computer systems with map assisted state transfer

Described is a system and method of regenerating stack unwind data in the presence of exceptions. The system is directed to generating metadata for use during stack unwinding. The system includes procedures, a first plurality of metadata, and an unwind rewriter. Each metadata is associated with a corresponding procedure in the procedures. The unwind rewriter generates new metadata from the first plurality of metadata in response to a modification of the sequence of binary instructions within a procedure, such that the new metadata accurately represents the modified sequence of binary instructions. The method regenerates metadata in response to a binary modification to a procedure by receiving a first group of metadata associated with the unmodified procedure, parsing the first group of metadata, and generating a second group of metadata that accurately represents the binary modification to the procedure.

Method and system for rewriting unwind data in the presence of exceptions

An architecture and implementation for losslessly restarting subsystems in a distributed file system is described. By partitioning functionality and logging appropriately across the kernel and user-level boundaries on a client, the user-level subsystem may be made losslessly restartable. Practical mechanisms for supporting state-based recovery in replicated state machines and like replica are described. In particular, each client daemon may include an operations log and an applied log sequence number. Each client driver may include a potentially different operations log. Each client daemon may be configured to request logged operations associated with log sequence numbers in one or more ranges specified by a specification that includes the applied log sequence number. The requested logged operations may reside in the operations log maintained by a client driver. Each client daemon may operate in accordance with user-level constraints and each client driver may operate in accordance with kernel-level constraints.

Lossless recovery for computer systems with remotely dependent data recovery

Described is a method that identifies a predicate expression representing conditions in predicated assembly language instructions that determine a direction of a conditional branch instruction. The predicate expression is employed to enable a transformation to be made that causes the conditional branch instruction to trigger, or execute, when an opposite condition is true. A method is directed to producing a binary-level conditional branch reversal within a binary program on a computer architecture that supports a predicated execution. The method includes obtaining a predicate expression representing a condition that influences a direction of program flow of the binary-level conditional branch to be reversed, determining a binary-level transformation that causes the binary-level conditional branch to be triggered when an opposite condition is true, and modifying the binary-level conditional branch with the determined binary-level transformation, wherein the binary-level conditional branch is reversed.

Growing community of inventors

Woodinville, WA, United States of America

Ronnie Chaiken