D-matrix Corporation

Agere Systems Inc.

Sudeep Bhoja

Siddharth Sheth

Aayush Ankit

Jayaprakash Balachandran

Akhil Arunkumar

Nithesh Kurella

Irene Quek

An AI accelerator apparatus using in-memory compute chiplet devices. The apparatus includes a first semiconductor substrate having a plurality of chiplets, each of which includes a plurality of tiles. Each tile includes a plurality of slices, a central processing unit (CPU), and a hardware dispatch device. Each slice can include a digital in-memory compute (DIMC) device configured to perform high throughput computations. In particular, the DIMC device can be configured to accelerate the computations of attention functions for transformer-based models (a.k.a. transformers) applied to machine learning applications. The chiplets are in a full mesh connectivity configuration such that at least one of the die-to-die (D2D) interconnects of each chiplet is coupled to one of the D2D interconnects of each other chiplet using a non-diagonal link. The chiplets can also include other interfaces to facilitate communication between the chiplets, memory and a server or host system.

AI accelerator apparatus using full mesh connectivity chiplet devices for transformer workloads

A server system with AI accelerator apparatuses using in-memory compute chiplet devices. The system includes a plurality of multiprocessors each having at least a first server central processing unit (CPU) and a second server CPU, both of which are coupled to a plurality of switch devices. Each switch device is coupled to a plurality of AI accelerator apparatuses. The apparatus includes one or more chiplets, each of which includes a plurality of tiles. Each tile includes a plurality of slices, a CPU, and a hardware dispatch device. Each slice can include a digital in-memory compute (DIMC) device configured to perform high throughput computations. In particular, the DIMC device can be configured to accelerate the computations of attention functions for transformer-based models (a.k.a. transformers) applied to machine learning applications. A single input multiple data (SIMD) device configured to further process the DIMC output and compute softmax functions for the attention functions.

Server system with AI accelerator apparatuses using in-memory compute chiplet devices for transformer workloads

An AI accelerator apparatus using in-memory compute chiplet devices. The apparatus includes one or more chiplets, each of which includes a plurality of tiles. Each tile includes a plurality of slices, a central processing unit (CPU), and a hardware dispatch device. Each slice can include a digital in-memory compute (DIMC) device configured to perform high throughput computations. In particular, the DIMC device can be configured to accelerate the computations of attention functions for transformer-based models (a.k.a. transformers) applied to machine learning applications. A single input multiple data (SIMD) device configured to further process the DIMC output and compute softmax functions for the attention functions. The chiplet can also include die-to-die (D2D) interconnects, a peripheral component interconnect express (PCIe) bus, a dynamic random access memory (DRAM) interface, and a global CPU interface to facilitate communication between the chiplets, memory and a server or host system.

AI accelerator apparatus using in-memory compute chiplet devices for transformer workloads

An AI accelerator apparatus using in-memory compute chiplet devices. The apparatus includes one or more chiplets, each of which includes a plurality of tiles. Each tile includes a plurality of slices, a central processing unit (CPU), and a hardware dispatch device. Each slice can include a digital in-memory compute (DIMC) device configured to perform high throughput computations. In particular, the DIMC device can be configured to accelerate the computations of attention functions for transformer-based models (a.k.a. transformers) applied to machine learning applications, including generative AI. A single input multiple data (SIMD) device configured to further process the DIMC output and compute softmax functions for the attention functions. The chiplet can also include die-to-die (D2D) interconnects, a peripheral component interconnect express (PCIe) bus, a dynamic random access memory (DRAM) interface, and a global CPU interface to facilitate communication between the chiplets, memory and a server or host system.

Generative AI accelerator apparatus using in-memory compute chiplet devices for transformer workloads

Ai accelerator apparatus using in-memory compute chiplet devices for transformer workloads

A method of encoding a plural-bit data word as a plurality of multi-level symbols, where each of the plurality of multi-level symbols has a value selected from a predetermined plurality of levels. The method includes first translating each one of the selected bit positions of the plural-bit data word to one of the levels. When the contents of a predetermined one of the bits of the data word is a predetermined value, the method provides a second translation of each of the selected bit positions of the plural-bit data word to one of the levels. The method further includes generating a plural-bit offset word from predetermined bit positions of the data word and generating the multi-level symbols by addition of the offset word to the translated levels. One embodiment of the invention provides that the multi-level symbols are assigned a five-level code and the codes are treated as twos-complement numbers.

Growing community of inventors

Santa Clara, CA, United States of America

Sudeep Bhoja