Microsoft Technology Licensing, LLC

Google Inc.

Paul Ronald Barham

Michael Isard

Joseph N Figueroa

Jeffrey Adgate Dean

Sanjay Ghemawat

Arthur Charles Tomlin

Yonghui Wu

Edmund Bernard Nightingale

Ashraf A Michail

Jeffrey Neil Margolis

Ruoming Pang

Brian LaMacchia

Chandramohan Amyangot Thekkath

Nathan Ackerman

Vijay Vasudevan

Martin Thomas Shetter

Aakanksha Chowdhery

Laurent El Shafey

Douglas Kevin Service

Daniel William Hurt

Parker Edward Schuh

Shahram Izadi

Alexandru Octavian Balan

Jason S Flaks

Ulfar Erlingsson

Edward Pearce Wobber

Georg Klein

Andrew Martin Pearson

Kevin Woo

Sudip Roy

Ryan Sepassi

Calvin Chan

Oliver Michael Christian Williams

Brennan Saeta

Hyeontaek Lim

Tuan Wong

David Kim

David G Molyneaux

Oliver M Williams

Thomas Michael Roeder

Jeremiah Spradlin

Oliver M C Williams

Steve Hodges

Otmar Hiliges

Ashraf Ayman Michail

Jeremiah C Spradlin

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for distributing machine learning workloads, e.g., computations for training a neural network or computing an inference using a neural network, across multiple hardware accelerators. One of the systems comprises a plurality of accelerator islands, each hardware accelerator island comprising a respective plurality of hardware devices that include a plurality of hardware accelerators and a corresponding host for each of the plurality of hardware accelerators; and a respective scheduler for each of the accelerator islands that is configured to schedule workloads across the plurality of accelerators and corresponding hosts in the accelerator island, wherein the system is configured to: receive data representing a machine learning workload; and assign a respective portion of the machine learning workload to each of the plurality of accelerator islands for scheduling by the respective scheduler for the accelerator island.

Asynchronous distributed data flow for machine learning workloads

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for receiving, by a computational graph system, a request to process a computational graph; obtaining data representing a subgraph of the computational graph, the computational graph comprising a plurality of nodes and directed edges, wherein each node represents a respective operation, wherein each directed edge connects a respective first node to a respective second node, the subgraph assigned to a first device by a placer in the computational graph system; determining that the first device comprises a hardware accelerator having a plurality of streams; in response to determining, generating instructions that when executed by the first device cause the first device to: assign the operation represented by each node in the subgraph to a respective stream; and perform the operations represented by the nodes in the subgraph in accordance with the assignment.

Stream-based accelerator processing of computational graphs

A performance monitoring system is described herein that works with a hypervisor to reserve resources outside of a virtual machine to monitor performance of an application running within the virtual machine. The application receives the guaranteed amount of resources specified by the virtual machine's specifications, and the monitoring consumes resources not associated with the virtual machine. The application running in the virtual machine is already isolated by the hypervisor from the physical machine, and thus additional burdens on the physical machine that allow the hypervisor to continue meeting guarantees to the virtual machine will not impact application performance. The performance monitoring system provides instrumentation of deployed applications that information technology (IT) personnel can dynamically turn on and off without affecting the performance of the deployed application. The performance monitoring system provides a robust framework for monitoring production applications without affecting the performance of those applications during execution.

Probe insertion via background virtual machine

Methods for generating and displaying images associated with one or more virtual objects within an augmented reality environment at a frame rate that is greater than a rendering frame rate are described. The rendering frame rate may correspond with the minimum time to render images associated with a pose of a head-mounted display device (HMD). In some embodiments, the HMD may determine a predicted pose associated with a future position and orientation of the HMD, generate a pre-rendered image based on the predicted pose, determine an updated pose associated with the HMD subsequent to generating the pre-rendered image, generate an updated image based on the updated pose and the pre-rendered image, and display the updated image on the HMD. The updated image may be generated via a homographic transformation and/or a pixel offset adjustment of the pre-rendered image.

Late stage reprojection

A data transfer system is described herein that allows data to be sent directly between two computing devices at the request of a third party client computer. The system allows a third party to initiate data transfers between computers in a network file system. This results in a significant speed increase because little to no data travels over the third party's potentially slower connection. The data transfer system provides a mechanism to determine if the direct transfer would be more efficient than two separate read and write operations, based on measurements of bandwidth and latency between each computing device. The data transfer system provides support for the source server to compress the data and the destination server to decompress the data at the direction of a third party client to further save network bandwidth.

Third party initiation of communications between remote parties

A computer system includes one or more field programmable gate arrays as a coprocessor that can be shared among processes and programmed using hardware libraries. Given a set of hardware libraries, an update process periodically updates the libraries and/or adds new libraries. One or more update servers can provide information about libraries available for download, either in response to a request or by notifying systems using such libraries. New available libraries can be presented to a user for selection and download. Requests for updated libraries can arise in several ways, such as through polling for updates, exceptions from applications attempting to use libraries, and upon compilation of application code.

Updating hardware libraries for use by applications on a computer system with an FPGA coprocessor

A personal datacenter system is described herein that provides a framework for leveraging multiple heterogeneous computers in a dynamically changing environment together as an ad-hoc cluster for performing parallel processing of various tasks. A home environment is much more heterogeneous and dynamic than a typical datacenter, and typical datacenter scheduling strategies do not work well for these types of small clusters. Machines in a home are likely to be powered on and off, be removed and taken elsewhere, and be connected by an ad-hoc network topology with a mix of wired and wireless technologies. The personal data center system provides components to overcome these differences. The system identifies a dynamically available set of machines, characterizes their performance, discovers the network topology, and monitors the available communications bandwidth between machines. This information is then used to compute an efficient execution plan for data-parallel and/or High Performance Computing (HPC)-style applications.

Scheduling and management in a personal datacenter

Field programmable gate arrays can be used as a shared programmable co-processor resource in a general purpose computing system. Components of an FPGA are isolated to protect the FPGA and data transferred between the FPGA and other components of the computer system. For example, data written by the FPGA to memory is encrypted, and is decrypted within the FPGA when read back from memory. Data transferred between the FPGA and other components such as the CPU or GPU, whether directly or through memory, can similarly be encrypted using cryptographic keys known to the communicating components. Transferred data also can be digitally signed by the FPGA or other component to provide authentication. Code for programming the FPGA can be encrypted and signed by the author, loaded into the FPGA in an encrypted state, and then decrypted and authenticated by the FPGA itself, before programming the FPGA with the code.

Managing use of a field programmable gate array with isolated components

A system and method for providing an augmented reality environment in which the environmental mapping process is decoupled from the localization processes performed by one or more mobile devices is described. In some embodiments, an augmented reality system includes a mapping system with independent sensing devices for mapping a particular real-world environment and one or more mobile devices. Each of the one or more mobile devices utilizes a separate asynchronous computing pipeline for localizing the mobile device and rendering virtual objects from a point of view of the mobile device. This distributed approach provides an efficient way for supporting mapping and localization processes for a large number of mobile devices, which are typically constrained by form factor and battery life limitations.

Distributed asynchronous localization and mapping for augmented reality

A system for providing augmented reality detects foreground occluders in an image of a video stream. One or more virtual objects are then rendered appropriately with respect to the occluders. Location information associated with the image is used to retrieve a three dimensional representation of the location where the image was taken. Features that are expected to appear in the image based on the three dimensional location but that cannot be located are used to determine regions of the image that are likely to include foreground occluders. Pixels in these regions are used to train a color model that classifies pixels as either part of the background of the image or part of one or more foreground occluders. The pixels in the image are classified using the model, and one or more virtual objects are rendered so that they appear behind any foreground occluders.

Determining foreground regions and background regions in an image

A secure location system is described herein that leverages location-based services and hardware to make access decisions. Many mobile computers have location devices, such as GPS. They also have a trusted platform module (TPM) or other security device. Currently GPS location data is made directly accessible to untrusted application code using a simple protocol. The secure location system provides a secure mechanism whereby the GPS location of a computer at a specific time can be certified by the operating system kernel and TPM. The secure location system logs user activity with a label indicating the geographic location of the computing device at the time of the activity. The secure location system can provide a difficult to forge, time-stamped location through a combination of kernel-mode GPS access and TPM security hardware. Thus, the secure location system incorporates secure location information into authorization and other operating system decisions.

Tamper proof location services

Growing community of inventors

San Francisco, CA, United States of America

Paul Ronald Barham