Microsoft Technology Licensing, LLC

Sudarshan Raghunathan

Adam P Jenkins

Richard A Warren

Damon Robert Hachmeister

Debi P Mishra

Tong Wen

Parry Husbands

Samuel Weiss

Andy Linfoot

Vijay Krishna Narayanan

Parry Jones Reginald Husbands

Ilya Matiach

Akshaya Annavajhala

Chun Sun

C David Callahan, Ii

Chang Young Park

Various embodiments herein each include at least one of systems, methods, and software for instantiating, executing, and operating dynamic hybrid computing environments, such as in cloud computing. Some such embodiments include allocating computing resources of a first server cluster to instantiate a first cluster and to establish a computing session. This embodiment may then initiate execution of a program within the first cluster that offloads at least one computing task to a second cluster, when the second cluster is instantiated, to leverage high-computing speed performance capabilities of the second cluster with regard to certain computing operations. Upon completion of program execution, the second cluster is then deallocated.

Dynamic hybrid computing environment

A platform that provides a way to automatically compose and execute even complex workflows without writing code is described. A set of pre-built functional building blocks can be provided. The building blocks perform data transformation and machine learning functions. The functional blocks have well known plug types. The building blocks can be composed build complex compositions. Input and output files are converted to a standard data type so that modules are pluggable.

Composing and executing workflows made up of functional pluggable building blocks

An interoperable platform that provides a way to automatically compose and execute even complex workflows without writing code is described. A set of pre-built functional building blocks can be provided. The building blocks perform data transformation and machine learning functions. The functional blocks have few well known plug types. The building blocks can be composed to build complex compositions. Interoperability between data formats, metadata schema and interfaces to machine learning (ML) functions and trained machine learning models can be provided with no loss of information. A cloud runtime environment can be provided in which the composed workflows can be hosted as REST API to run in production.

Interoperable machine learning platform

A sparse dataset structure is created by creating column vectors for one or more columns in a dataset that have at least one significant value. Each column vector includes data values for columns of the dataset. Each column vector that is a sparse column vector includes a look-up index array and a value array. Entries in the look-up index array represent columns. The value array includes values for a row in a column. Each entry in the value array points to a row entry in the look-up index array. A side structure includes a row index and a column index. The row index includes a location for an entry for each row where entries point to a location in the column index that identifies a column that has a first significant entry for a row. Alternatively a sparse dataset could be constructed with sparse rows.

Sparse datatable data structure

Object serialization is used to communicate references regarding shim objects. Shim objects are instantiated on one or more ranks of a distributed software application. The shim objects store a registration object in a distributed object cache for each rank. The registration object includes a unique identifier for a distributed array object and a reference to a local portion of the distributed array. The shim objects are serialized for communication of the stored references from a master rank of the distributed application to one or more worker ranks of the distributed application. Upon serializing the shim objects, the shim object's stored references are communicated from the distributed object cache for that rank to the one or more worker ranks of the distributed application. The shim objects are subsequently removed so that references to the underlying distributed array object are also removed, and memory previously allocated to the unique identifier is recoverable.

Providing distributed array containers for programming objects

Computerized singular value decomposition of an input complex matrix. A real-value matrix representation of the input complex matrix is provided to a singular value decomposition module, which correctly obtains a singular value representation of the real-value matrix representation. However, the result is not provided in a form for convenient conversion back into a valid singular value decomposition solution for the original input complex matrix, as the upper left half and lower right half of the diagonal of the diagonal matrix are not identical. A correction module corrects by formulating a corrected diagonal matrix that represents the value of the diagonal of the first diagonal matrix, but shuffled so that the upper left half of the diagonal of the second diagonal matrix is the same as the lower right half of the diagonal of the second diagonal matrix. Corrected unitary matrices may also be formed.

Singular value decomposition of complex matrix

Embodiments are directed to establishing registration objects for distributed processes, to managing memory on worker processes of a distributed software application and to using object serialization to communicate references to shim objects. In one scenario, a computer system accesses distributed process instances in a distributed runtime and creates a registration object for each of the process instances in the distributed runtime. The registration object includes a key value pair, where the key includes a unique identifier (ID) that identifies a distributed array instance associated with the distributed process, and the value includes a reference to a local portion of the distributed array instance. The computer system then maintains a mapping between the unique ID and the distributed array instance using the registration object. As such, the key value refers to the local portion of the same distributed array instance on each distributed process of the distributed runtime.

Embodiments are directed to using object serialization to communicate references to shim objects and to managing memory on worker processes of a distributed software application. In one scenario, a computer system instantiates shim objects on one or more ranks of a distributed application. The shim objects are configured to store a reference to an instance of a native partitioned array, where the reference includes a unique identifier for the native partitioned array instance. The computer system then serializes the shim objects for communication of the stored references from the master rank of the distributed application to various other worker ranks of the distributed application. Then, upon serializing the shim objects, the computer system communicates the shim object's stored references to the other worker ranks of the distributed application.

Partitioned array objects in a distributed runtime

Embodiments are directed to providing a generic software function for a plurality of data types in a distributed environment and to implementing type checking at compile time and at runtime. In one scenario, a computer system receives a portion of software code for a distributed software application, where the software code is written as a generic function in a computer programming language that supports reflection. The computer system presents a function signature at runtime to a just-in-time (JIT) compiler. The function signature describes which of multiple different input and output data types the generic function is configured to work with. The computer system also implements reflection at runtime to resolve each generic function to a function pointer after compilation by the JIT.

Implementation of distributed methods that support generic functions

A serial testing infrastructure includes the capability to execute a distributed test on multiple virtual processors. A test executable may be stored in a library and the test description, including the name of the test, the test library, and other test characteristics, may be stored in a separate test data file. The serial testing infrastructure initiates multiple distributed test executors that each launch an instance of the distributed test as a process that runs concurrently with other instances of the distributed test. Each distributed test executor monitors execution of it corresponding process until completion or timeout.

Distributed testing within a serial testing infrastructure

Embodiments are directed to reshaping a partitioned data array. In an embodiment, a computer system identifies a block length parameter that describes the number of data blocks in the range-partitioned flattened representation of the array that appear consecutively in each locale. The computer system then identifies a stride parameter that describes the amount of separation between data blocks in the range-partitioned flattened representation of the array that appear consecutively in a plurality of locales. Based on the identified block length parameter and the stride parameter, the computer system determines which of the data blocks on the plurality of locales are to be sent to other locales to produce a local version of the reshaped array. The computer system then receives data blocks from the different locales in the distributed system and reconstructs the array based on the received blocks to create a local version of the reshaped array.

Global distributed memory reshape operations on range-partitioned arrays

Embodiments are directed to handling errors in a distributed software application. In one scenario, a computer system instantiates a distributed application which is configured for execution on multiple different computer systems. The computer system determines that an exception has occurred during the execution of the distributed application on at least one of the computer systems and translates the exception that occurred during distributed execution in a distributed environment into a serial exception that appears to have occurred during serial execution in a serial execution environment. The computer system then presents the translated serial exception to an application developer or other user.

Growing community of inventors

Cambridge, MA, United States of America

Sudarshan Raghunathan