Google Inc.

Dehao Chen

Xinliang David Li

Vinodha Ramasamy

Zhifeng Chen

Robert Hundt

Peng Yuan

Jiquan Ngiam

Yanping Huang

Tipp Moseley

Youlong Cheng

HyoukJoong Lee

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training giant neural networks. One of the methods includes obtaining data specifying a partitioning of the neural network into N composite layers that form a sequence of composite layers, wherein each composite layer comprises a distinct plurality of layers from the multiple network layers of the neural network; obtaining data assigning each of the N composite layers to one or more computing devices from a set of N computing devices; partitioning a mini-batch of training examples into a plurality of micro-batches; and training the neural network, comprising: performing a forward pass through the neural network until output activations have been computed for each micro-batch for a final composite layer in the sequence, and performing a backward pass through the neural network until output gradients have been computed for each micro-batch for the first composite layer in the sequence.

Training giant neural networks using pipeline parallelism

A system and method for improving the performance of all applications are disclosed. Production profile data may be collected about each application while the application is executing. The production profile data may be converted into symbolized profiles and stored in a database. The symbolized profiles may be aggregated into a single aggregated profile. This aggregated profile may be used as a compilation input when compiling new versions of an application's binary to improve the application's performance for observed application behavior.

Continuous profiling for automatic feedback directed optimization

A system and method for using inline stacks to improve the performance of application binaries is included. While executing a first application binary, profile data may be collected about the application that includes which callee functions are called from the application's callsites and the number of times each inline stack is executed. A context summary map may be created from the collected profile data which shows a summary of the total execution count of all instructions in the callee function for each callsite inlined in the application's normal binary. Using the context summary map, each function callsite's execution count may be compared with a predetermined threshold to determine if the function should be inlined. Then the application's profile may be annotated and a second application binary, an optimized binary, may be generated using the annotated profile.

Using an inline stack to improve performance of an applications binary

While optimizing executable code, compilers traditionally make static determinations about whether or not to inline functions. Embodiments of the invention convert dynamic hardware-event sampling information into context-specific edge frequencies, which can be used to make inlining decisions for functions.

Methods for handling inlined functions using sample profiles

Traditional feedback-directed optimization (FDO) is not widely used due to the significant computational overhead involved in using instrumented binaries. The described embodiments provide methods that eliminate the need for instrumented binaries by permitting the conversion of hardware-event sampling information into edge frequencies usable by FDO compilers. Some advantages include: the ability to collect feedback data on production systems; the ability to perform FDO on the OS kernel; and the ability to avoid disrupting timing paths from instrumented binaries.

Growing community of inventors

Mountain View, CA, United States of America

Dehao Chen