Microsoft Technology Licensing, LLC

Yifung Lin

Omid Afnan

Dian Zhang

Jin Sun

Na Gao

Zhaoji Chen

Deyang Song

David Joseph Cummings

Scott Morrison

Régis Lucien Francis Brid

Michael Furman

A computerized mechanism to formulate a data flow representation from a syntax tree. The syntax tree may be, for instance, an abstract syntax tree (AST) that is formulated by compiling query script, such as big data query script. Each node in the syntax tree is composed of one or more tokens (script portions) that are taken from the query script, which relationships between the tokens being syntactically represented by links between those tokens. Accordingly, the data flow representation may also be a data flow representation of the original query script itself. In order to formulate the data flow representation from the syntax tree, the data types of the various inputs and outputs of the syntax tree nodes are identified and bound to a corresponding data flow. This may be performed whilst honoring dependencies between the nodes. The data flow may be visualized to an author or reviewer of the script.

Generation of data flow from syntax tree

Reducing data skew when performing computational jobs (such as responding to complex queries or queries on big data sets). To gather performance information for a vertex in a distributed stage, the distributed stage is processed and the resulting performance information being generated. The system then identifies a performance outlier of the vertices, which varies from a performance norm of other vertices. A trigger instruction is then generated that is structured to trigger the performance outlier vertex to gather data distribution information of the data that the performance outlier vertex previously processed. The trigger instruction may thus be executed to cause the performance outlier vertex to gather the data distribution information for evaluation for data skew.

Data skew finding and analysis

Simulation of a computational job using various sets of resources, and potentially also the automated or semi-automated allocation of an appropriate set of resources for accomplishing a computational job comprising multiple vertices. For each of multiple potential sets of resources, a simulation module simulates processing of the computational job. While the simulation does not actually perform the vertex on each processing node, the simulation does use dependencies between vertices, and historical data regarding the processing of instances of such vertices, in order to determine the efficacy of processing of each vertex, and to put the estimations together into an overall simulation result.

Modeling resource usage for a job

The supporting of debugging of failed vertex code on a test machine. This debugging is made possible even though the vertex code failed while on a remote processing node, and is but one of multiple, and potentially innumerable vertices that run in a distributed environment. This represents a vast technical improvement over prior ad hoc methods for trying to debug a large distributed application, since time is not wasted on vertices that operated properly, but rather debugging is focused on the problem vertex. Even reproducing the failure is a huge technical step forward.

Debugging remote vertex code on test machine

A DataGrid user interface (UI) control is presented that is configured to display a table that includes columns corresponding to fields in a data source and rows corresponding to records in the data source. The DataGrid UI control is also configured to display a details section associated with each row in the table. One or all of the details sections can be displayed or hidden based upon properties specified by a developer or input made by a user. A property may also be set to cause the details section for a row to be displayed only when a user interface cursor is hovering over the row associated with the details section. A property may also be set to cause a details section for a row to be displayed only when the associated row is selected.

Growing community of inventors

Beijing, China

Yifung Lin