International Business Machines Corporation

Globalfoundries Inc.

Tsinghua University

Chia-Yu Chen

Tenko Yamashita

Zuoguang Liu

Kailash Gopalakrishnan

Jungwook Choi

Vijayalakshmi Srinivasan

Jui-Hsin Lai

Bruce Bennett Doris

Hong He

Rajasekhar Venigalla

Li-Wen Hung

Sanjay C Mehta

Pin-Yu Chen

Jintao Zhang

Ko-Tao Lee

Naigang Wang

Lingfei Wu

Miaomiao Wang

Sunil K Shukla

Jinwook Oh

Victor Han

Chun-chen Yeh

Shu-Jen Han

Ankur Agrawal

Sijia Liu

Suyog Gupta

Xiao Sun

Damon B Farmer

Steven J Bentley

Bruce Martin Fleischer

Brian William Curran

Swagath Venkataramani

Yinglong Xia

Pei-Yun S Hsueh

Mingu Kang

Pierce I-Jen Chuang

Ajey Poovannummoottil Jacob

Kyu-hyoun Kim

Wei Zhang

Seyoung Kim

Xiao Lin Sun

Michael John Hargrove

Yada Zhu

Pritish Narayanan

Ryan O Jung

Daniel Brand

I-Hsin Chung

Andrea Fasoli

Kyu-Hyoun Kim

Kun Xu

Kailash Gopalakrishan

Lee M Saltzman

Sivanandha K Kanakasabapathy

Jiamin Ni

Tsung-Yi Ho

Yun-Yun Tsai

An apparatus includes a floating-point gradient register; an integer register; a memory bank; and an array of processing units. Each of the units includes a plurality of binary shifters having an integer input configured to obtain corresponding bits of a 4-bit integer multiplicand, and a shift-specifying input configured to obtain corresponding bits in an exponent field of a 4-bit floating point multiplier. The multiplier is specified in a mantissaless four-bit floating point format including a sign bit, three exponent bits, and no mantissa bits. An adder tree has a plurality of inputs coupled to outputs of the plurality of shifters, and a rounder has an input coupled to an output of the adder tree. The integer inputs are connected to the integer register; the shift-specifying inputs are connected to the floating-point gradient register; and outputs of the rounders are coupled to the memory bank.

Four-bit training for machine learning

An apparatus includes multiple analog to digital converters. Individual analog to digital converters are configured to produce a digital output from an analog input and configured to compute a least significant bit of the digital output by comparing an internal residual voltage for determination of the least significant bit and a residual voltage from another analog to digital converter.

Stochastic rounding switched capacitor computation cores useful for efficient deep learning inference

An apparatus includes circuitry for a neural network that is configured to perform forward propagation neural network operations on floating point numbers having a first n-bit floating point format. The first n-bit floating point format has a configuration consisting of a sign bit, m exponent bits and p mantissa bits where m is greater than p. The circuitry is further configured to perform backward propagation neural network operations on floating point numbers having a second n-bit floating point format that is different than the first n-bit floating point format. The second n-bit floating point format has a configuration consisting of a sign bit, q exponent bits and r mantissa bits where q is greater than m and r is less than p.

Neural network circuitry having floating point format with asymmetric range

An adversarial robustness testing method, system, and computer program product include testing, via an accelerator, a robustness of a black-box system under different access settings, where the testing includes tearing down the robustness testing to a subtask of a predetermined size.

Testing adversarial robustness of systems with limited access

An apparatus for training and inferencing a neural network includes circuitry that is configured to generate a first weight having a first format including a first number of bits based at least in part on a second weight having a second format including a second number of bits and a residual having a third format including a third number of bits. The second number of bits and the third number of bits are each less than the first number of bits. The circuitry is further configured to update the second weight based at least in part on the first weight and to update the residual based at least in part on the updated second weight and the first weight. The circuitry is further configured to update the first weight based at least in part on the updated second weight and the updated residual.

Machine learning hardware having reduced precision parameter components for efficient parameter update

A method for improving performance of a predefined Deep Neural Network (DNN) convolution processing on a computing device includes inputting parameters, as input data into a processor on a computer that formalizes a design space exploration of a convolution mapping, on a predefined computer architecture that will execute the predefined convolution processing. The parameters are predefined as guided by a specification for the predefined convolution processing to be implemented by the convolution mapping and by a microarchitectural specification for the processor that will execute the predefined convolution processing. The processor calculates performance metrics for executing the predefined convolution processing on the computing device, as functions of the predefined parameters, as proxy estimates of performance of different possible design choices to implement the predefined convolution processing.

Method to map convolutional layers of deep neural network on a plurality of processing elements with SIMD execution units, private memories, and connected as a 2D systolic processor array

Transfer learning in machine learning can include receiving a machine learning model. Target domain training data for reprogramming the machine learning model using transfer learning can be received. The target domain training data can be transformed by performing a transformation function on the target domain training data. Output labels of the machine learning model can be mapped to target labels associated with the target domain training data. The transformation function can be trained by optimizing a parameter of the transformation function. The machine learning model can be reprogrammed based on input data transformed by the transformation function and a mapping of the output labels to target labels.

Transfer learning with machine learning systems

A system, having a memory that stores computer executable components, and a processor that executes the computer executable components, reduces data size in connection with training a neural network by exploiting spatial locality to weight matrices and effecting frequency transformation and compression. A receiving component receives neural network data in the form of a compressed frequency-domain weight matrix. A segmentation component segments the initial weight matrix into original sub-components, wherein respective original sub-components have spatial weights. A sampling component applies a generalized weight distribution to the respective original sub-components to generate respective normalized sub-components. A transform component applies a transform to the respective normalized sub-components. A cropping component crops high-frequency weights of the respective transformed normalized sub-components to yield a set of low-frequency normalized sub-components to generate a compressed representation of the original sub-components.

Compression of fully connected / recurrent layers of deep network(s) through enforcing spatial locality to weight matrices and effecting frequency compression

Techniques that facilitate graph similarity analytics are provided. In one example, a system includes an information component and a similarity component. The information component generates a first information index indicative of a first entropy measure for a first graph-structured dataset associated with a machine learning system. The information component also generates a second information index indicative of a second entropy measure for a second graph-structured dataset associated with the machine learning system. The similarity component determines similarity between the first graph-structured dataset and the second graph-structured dataset based on a graph similarity computation associated with the first information index and the second information index.

Graph similarity analytics

An adversarial robustness testing method, system, and computer program product include testing a robustness of a black-box system under different access settings via an accelerator.

Systems, computer-implemented methods, and computer program products to facilitate gradient weight compression are provided. According to an embodiment, a system can comprise a memory that stores computer executable components and a processor that executes the computer executable components stored in the memory. The computer executable components can comprise a pointer component that can identify one or more compressed gradient weights not present in a first concatenated compressed gradient weight. The computer executable components can further comprise a compression component that can compute a second concatenated compressed gradient weight based on the one or more compressed gradient weights to update a weight of a learning entity of a machine learning system.

Asynchronous gradient weight compression

A Static Random Access Memory (SRAM) device in a binary neural network is provided. The SRAM device includes an SRAM inference engine having an SRAM computation architecture with a forward path that include multiple SRAM cells forming a chain of SRAM cells such that an output of a given one of the multiple SRAM cells is an input to a following one of the multiple SRAM cells. The SRAM computation architecture is configured to compute a prediction from an input. The SRAM computation architecture is configured to store ternary data and perform local computations on the ternary data.

Inference focus for offline training of SRAM inference engine in binary neural network

An apparatus includes a memory and a processor coupled to the memory. The processor includes first and second sets of arithmetic units having first and second precision for floating-point computations, the second precision being lower than the first precision. The processor is configured to obtain a machine learning model trained in the first precision, to utilize the second set of arithmetic units to perform inference on input data, to utilize the first set of arithmetic units to generate feedback for updating parameters of the second set of arithmetic units based on the inference performed on the input data by the second set of arithmetic units, to tune parameters of the second set of arithmetic units based at least in part on the feedback generated by the first set of arithmetic units, and to utilize the second set of arithmetic units with the tuned parameters to generate inference results.

Mixed precision capable hardware for tuning a machine learning model

A Static Random Access Memory (SRAM) device in a binary neural network is provided. The SRAM device includes an SRAM inference engine having an SRAM computation architecture with a forward path that include multiple SRAM cells. The multiple SRAM cells are configured to form a chain of SRAM cells such that an output of a given one of the multiple SRAM cells is an input to a following one of the multiple SRAM cells. The SRAM computation architecture is configured to compute a prediction from an input.

Methods, systems, and circuits for training a neural network include applying noise to a set of training data across wordlines using a respective noise switch on each wordline. A neural network is trained using the noise-applied training data to generate a classifier that is robust against adversarial training.

Integrated noise generation for adversarial training

Growing community of inventors

White Plains, NY, United States of America

Chia-Yu Chen