Microsoft Technology Licensing, LLC

Microsoft Corporation

James Kelsey Howey

Praveen Seshadri

Ciprian Gerea

Michiel Johannes Wories

Michael A Blaszczak

Hongfei Guo

James A Holt

Theodore Tak Lee

Daniel T Jones

Grigory Pogulsky

Sethu Srinivasan

Sergei Ivanov

J Kirk Haselden

Jeffrey Bernhardt

Oscar Gerardo Naim

Slobodan M Bojanic

Jeff R Bernhardt

Anjan Kumar Das

Mark Durley

Ranjeeta Nanda

Maciek Sarnowicz

Vipul M Shah

The data transformation system in one embodiment, comprises a capability to receive data, a data destination and a capability to store transformed data, and a data transformation pipeline that constructs complex end-to-end data transformation functionality by pipelining data flowing from one or more sources to one or more destinations through various interconnected nodes for transforming the data as it flows. Each component in the pipeline possesses predefined data transformation functionality, and the logical connections between components define the data flow pathway in an operational sense. The data transformation pipeline enables a user to develop complex end-to-end data transformation functionality by graphically describing and representing, via a GUI, a desired data flow from one or more sources to one or more destinations through various interconnected nodes (graph). Each node in the graph selected by the user represents predefined data transformation functionality, and connections between nodes define the data flow pathway.

Systems and methods for scheduling data flow execution based on an arbitrary graph describing the desired data flow

User productivity can be facilitated by providing an interface between a high-level intent and low-level settings that should be configured to achieve the intent. The intent can be expressed in declarative language. The intent can be verified to determine if it conforms to a target, a policy, or both. Policy binding can provide a relationship between the target and the policy. An application view can be provided that allows the user to view a server having one or more databases.

Declarative management framework

The data transformation system in one embodiment, comprises a capability to receive data, a data destination and a capability to store transformed data, and a data transformation pipeline that constructs complex end-to-end data transformation functionality by pipelining data flowing from one or more sources to one or more destinations through various interconnected nodes for transforming the data as it flows. Each component in the pipeline possesses predefined data transformation functionality, and the logical connections between components define the data flow pathway in an operational sense. The data transformation pipeline enables a user to develop complex end-to-end data transformation functionality by graphically describing and representing, via a GUI,a desired data flow from one or more sources to one or more destinations through various interconnected nodes (graph). Each node in the graph selected by the user represents predefined data transformation functionality, and connections between nodes define the data flow pathway.

Systems, devices, and methods for modularizing data flows and for providing stand-alone execution of incomplete data flows are provided based on an improved data warehousing framework. Data flow modularization and code reuse is improved through the use of the disclosed flowlets and allows for consistent and efficient standalone testing of incomplete data flows. The disclosed details enable various refinements and modifications according to system design considerations.

Systems and methods for modularizing data flows

Automated sampling and aggregation of performance or other dynamically varying data facilitates user productivity by making data available for a wide range of uses. An automated sampling and aggregration system includes one or more types of data collectors, including a dedicated data collector per machine, multiple data collectors per machine, or an instance-less data collector.

Extensible data collectors

Systems, methods, and computer readable media for designing and executing workflows can be equipped with a means for manipulation of workflow steps by grouping them, such as by placing them in containers. A container can be associated with a database transaction. In some embodiments, a container property can be set to indicate whether the container must participate, may participate, or may not participate in a database transaction. Further, containers may be placed in parent containers, and optionally inherit the transaction property of the parent.

Systems and methods for controlling transaction participation for groups of steps in a workflow

The operation of a data flow is improved by providing for validation of the consistency of the data flow with the external data files it works with, even in the face of changes in those external data files Metadata is stored regarding the external data files. Because external data files may change, this metadata is used when determining whether the external data files are still compatible with the data flow. In such performing such validation, the metadata tracks what changes in the external data files have occurred, and allows increased flexibility in presenting the user with options or in automatically fixing the data flow to correspond with changes in external data files. When the external data files are not available, the stored metadata can be used to validate the data flow to the last information regarding the external data files. The external metadata can be resynchronized to the data flow or to the external data files to provide updated external metadata for later validation.

External metadata processing

Growing community of inventors

Seattle, WA, United States of America

James Kelsey Howey