Microsoft Technology Licensing, LLC

Omid Afnan

Dian Zhang

Pu Li

Yifung Lin

Jiahui Wang

Yaron Burd

Jin Sun

Na Gao

The visualization of progress of a distributed computational job at multiple points of execution. After a computational job is compiled into multiple vertices, and then those multiple vertices are scheduled on multiple processing nodes in a distributed environment, a processing gathering module gathers processing information regarding processing of multiple vertices of a computational job, and at multiple instances in time in the execution of the computational job. A user interface module graphically presents a representation of an execution structure representing multiple nodes of the computational job, and dependencies between the multiple nodes, where the nodes may be a single vertex or a group of vertices (such as a stage).

Progress visualization of computational job

Reducing data skew when performing computational jobs (such as responding to complex queries or queries on big data sets). To gather performance information for a vertex in a distributed stage, the distributed stage is processed and the resulting performance information being generated. The system then identifies a performance outlier of the vertices, which varies from a performance norm of other vertices. A trigger instruction is then generated that is structured to trigger the performance outlier vertex to gather data distribution information of the data that the performance outlier vertex previously processed. The trigger instruction may thus be executed to cause the performance outlier vertex to gather the data distribution information for evaluation for data skew.

Data skew finding and analysis

Comparing recurring processes. A method includes automatically identifying a plurality of recurring process instances having one or more commonalities. The method further includes displaying the plurality of recurring process instances having one or more commonalities to a user in a user interface. The method further includes receiving user input at the user interface selecting a first recurring process instance from the plurality of recurring process instances. The method further includes receiving user input at the user interface selecting a second recurring process instance from the plurality of recurring process instances. The method further includes automatically identifying differences in the first and second recurring process instances. The method further includes presenting the differences in the graphical user interface to the user.

Analysis of recurring processes

The supporting of debugging of failed vertex code on a test machine. This debugging is made possible even though the vertex code failed while on a remote processing node, and is but one of multiple, and potentially innumerable vertices that run in a distributed environment. This represents a vast technical improvement over prior ad hoc methods for trying to debug a large distributed application, since time is not wasted on vertices that operated properly, but rather debugging is focused on the problem vertex. Even reproducing the failure is a huge technical step forward.

Growing community of inventors

Beijing, China

Omid Afnan