International Business Machines Corporation

Yulong Li

Paul Michael Solomon

Effendi Leobandung

Siyuranga O Koswatta

Xiao Lin Sun

Chun-Chen Yeh

Dennis Merton Newns

Jin-Ping Han

Takashi Ando

Chun-chen Yeh

Guy M Cohen

Seyoung Kim

Nanbo Gong

Malte Johannes Rasch

Tak H Ning

Cedric Lichtenau

Vijayalakshmi Srinivasan

Kailash Gopalakrishnan

Tayfun Gokmen

Bishwaranjan Bhattacharjee

Zhibin Ren

Wei Wang

Swagath Venkataramani

Sunil K Shukla

Razvan Peter Figuli

Michele Merler

Holger Horbach

Masayasu Muraoka

Martin A Lutz

Yang Zhao

Graeme William Blackwood

Systems, computer-implemented methods, and computer program products to facilitate improving a mono-lingual language model using pre-trained models and parallel data are provided. According to an embodiment, a system can comprise a processor that executes components stored in memory. The computer executable components can comprise a training component that can use a first language model to improve a mono-lingual second language model through mask language modeling (MLM) training and contrastive learning. The computer executable components can comprise a data component that can collect parallel data for the first language model and the mono-lingual second language model. The computer executable components can comprise a tokenizing component that can tokenize the parallel data for the first language model and the mono-lingual second language model. The computer executable components can comprise an alignment component that can create an alignment of the tokenized parallel data.

Mono-lingual language models using parallel data

Technical solutions are described for storing weight in a crosspoint device of a resistive processing unit (RPU) array. An example system includes a crosspoint array, wherein each array node represents a connection between neurons of the neural network, and wherein each node stores a weight assigned to the node. The crosspoint array includes a crosspoint device at each node. The crosspoint device includes a counter that has multiple single bit counters, and states of the counters represent the weight to be stored at the crosspoint device. Further, the crosspoint device includes a resistor device that has multiple resistive circuits, and each resistive circuit is associated with a respective counter from the counters. The resistive circuits are activated or deactivated according to a state of the associated counter, and an electrical conductance of the resistor device is adjusted based at least in part on the resistive circuits that are activated.

Counter based resistive processing unit for programmable and reconfigurable artificial-neural-networks

Processing input data for transmittal to a data consumer such as an artificial intelligence engine is performed by arranging the input data into a uniform structure made up of sticks of data combined to form pages of sticks. A stick is any well-sized set of input data elements whereby the size of the stick is fixed. A masking pattern is established for sticks of data having certain ranges of invalid data for consumption of partial sticks while maintaining validity of the input data being transferred. The mask pattern is derived based on set-active-mask-and-value (SAMV) instructions. The derived mask pattern is carried forward for subsequent load instructions to the data consumer.

Padding input data for artificial intelligence accelerators

A method and resulting structures for a semiconductor device includes forming a source terminal of a semiconductor fin on a substrate. An energy barrier is formed on a surface of the source terminal. A channel is formed on a surface of the energy barrier, and a drain terminal is formed on a surface of the channel. The drain terminal and the channel are recessed on either sides of the channel, and the energy barrier is etched in recesses formed by the recessing. The source terminal is recessed using timed etching to remove a portion of the source terminal in the recesses formed by etching the energy barrier. A first bottom spacer is formed on a surface of the source terminal and a sidewall of the semiconductor fin, and a gate stack is formed on the surface of the first bottom spacer.

Field effect transistor with controllable resistance

An electronic circuit includes a plurality of word lines; a plurality of bit lines intersecting said plurality of word lines at a plurality of grid points; and a plurality of in-memory processing cells located at said plurality of grid points. Each of said in-memory processing cells includes a first switch having a first terminal coupled to a corresponding one of said word lines and a second terminal; a second switch having a first terminal coupled to said second terminal of said first switch and a second terminal coupled to a corresponding one of said bit lines; and a non-volatile tunable capacitor having one electrode coupled to said second terminal of said first switch and said first terminal of said switch, and having another electrode coupled to ground.

Using ferroelectric field-effect transistors (FeFETs) as capacitive processing units for in-memory computing

In an approach for forming a nonvolatile tunable capacitor device, a first electrode layer is formed distally opposed from a second electrode layer, the first electrode layer configured to make a first electrical connection and the second electrode layer configured to make a second electrical connection. A dielectric layer is posited between the first electrode layer and adjacent to the second electrode layer. A phase change material (PCM) layer is posited between the first electrode layer and the second electrode layer adjacent to the dielectric layer. An energizing component is provided to heat the PCM layer to change a phase of the PCM layer. The energizing component may include a heating element or electrical probe in direct contact with the PCM layer, that when energized is configured to apply heat to the PCM layer. The phase of the PCM layer is changeable between an amorphous phase and a crystalline phase.

Nonvolatile tunable capacitive processing unit

A method for multiple copies of a set of multi-kernel set operations in a hardware accelerated neural network includes a word line for receiving a pixel value of an input image. A bit line communicates a modified pixel value. An analog memory cell including a first capacitor stores a first kernel weight of a first kernel in one of a plurality of kernel sets such that the pixel value is operated on by the first kernel weight to produce the modified pixel value. A charge connection connects the first capacitor to at least a second capacitor storing a second kernel weight of a related kernel of a second one of the plurality of kernel sets such that charge is shared between the first capacitor and at least the second capacitor to normalize the first kernel weight and the second kernel weight.

Kernel sets normalization with capacitor charge sharing

Methods and systems of implementing a convolutional neural network are described. In an example, a structure may receive input signals and distribute the input signals to a plurality of unit cells. The structure may include a plurality of multi-kernel modules that may include a respective set of unit cells. A unit cell may correspond to an element of a kernel being implemented in the convolutional neural network and may include a storage component configured to store a weight of a corresponding element of the kernel. A first pass gate of the unit cell may be activated to pass a stored weight of the unit cell to a plurality of operation circuits in the corresponding unit cell, such that the stored weight may be applied to the input signals. The structure may generate a set of outputs based on the application of the stored weights to the input signals.

Multi-kernel configuration for convolutional neural networks

Technical solutions are described for storing weight in a crosspoint device of a resistive processing unit (RPU) array. An example method includes setting a state of each single bit counter from a set of single bit counters in the crosspoint device, the states of the single bit counters representing the weight to be stored at the crosspoint device. The method further includes adjusting electrical conductance of a resistor device of the crosspoint device. The resistor device includes a set of resistive circuits, each resistive circuit associated with a respective single bit counter from the set of single bit counters, the electrical conductance adjusted by activating or deactivating each resistive circuit according to a state of the associated single bit counter.

Systems and methods for a capacitor based resistive processing unit with symmetrical weight updating include a first capacitor that stores a charge corresponding to a weight value. A readout circuit reads the charge stored in the first capacitor to apply a weight to an input value corresponding to an input signal using the weight value to produce an output. An update circuit updates the weight value stored in the first capacitor, including a second capacitor in communication with the first capacitor to transfer an amount of charge to the first capacitor according to an error of the output by changing a voltage difference across the first capacitor by a voltage change corresponding to the amount of charge, the voltage difference corresponding to the charge stored in the first capacitor.

Capacitor based resistive processing unit with symmetric weight update

A CMOS-based resistive processing unit (RPU) for a neural network. The RPU includes a capacitor device configured to store a charge representing a weight value associated with a neural network circuit operation. A current source Field Effect Transistor (FET) device is operatively connected to the capacitor device for increasing a charge on the capacitor. A current sink FET device operatively connected to the capacitor device is configured for decreasing the stored capacitor charge. An analog weight update circuit receives one or more update signals generated in conjunction with the neural network circuit operation, the analog weight update circuit controlling the current source FET device and the current sink FET device to provide either a determined amount of current to increase the stored charge on the capacitor device, or sink a determined amount of current to decrease the stored charge on the capacitor device.

Circuit for CMOS based resistive processing unit

A CMOS-based resistive processing unit (RPU) and method for a neural network. The RPU includes a capacitor device configured to store a charge representing a weight value associated with a neural network circuit operation. A current source Field Effect Transistor (FET) device is operatively connected to the capacitor device for increasing a charge on the capacitor. A current sink FET device operatively connected to the capacitor device is configured for decreasing the stored capacitor charge. An analog weight update circuit receives one or more update signals generated in conjunction with the neural network circuit operation, the analog weight update circuit controlling the current source FET device and the current sink FET device to provide either a determined amount of current to increase the stored charge on the capacitor device, or sink a determined amount of current to decrease the stored charge on the capacitor device.

Growing community of inventors

Hartsdale, NY, United States of America

Yulong Li