Xilinx, Inc.

Koninklijke Philips Corporation N.v.

Trident Microsystems (far East) Ltd.

Kornelis Antonius Vissers

Michaela Blott

Ralph D Wittig

Juan J Noguera Serra

Baris Ozgul

Jan Langer

Richard Lewis Walke

Kristof Denolf

Paul R Schumacher

Goran H K Bilski

Christopher H Dick

Philip Bryn James-Roxby

Abraham Karel Riemens

Robert Jan Schutten

Ling Liu

Kimon Karras

Tim Tuan

Jack S Lo

David Clarke

Nicholas Fraser

Robert D Turney

Goran Hk Bilski

Olukayode Anthony Ojo

Peter McColgan

Sunny C Huang

Mark Paluszkiewicz

Frans Sijstermans

Jos Van Eijndhoven

Jeroen Maria Kettenis

Lakshmi Rao

Marcel J A Tromp

Rudolf H J Bloks

Yaman Umuroglu

Louis Coulon

Giulio Gambardella

Ussama Zahid

Alireza Khodamoradi

Thomas Janson

David A Sidler

Raymond Carley

Thomas B English

Embodiments herein describe a learnable transform block disposed before, or in between, the neural network layers to transform received data into a more computational-friendly domain while preserving discriminative features required for the neural network to generate accurate results. In one embodiment, during a training phase, an AI system learns parameters for the transform block that are then used during the inference phase to transform received data into the computational-friendly domain that has a reduced size input. The transformed data may require less compute resources or less memory usage to process by the underlying hardware device that hosts the neural network.

Implementation-tuned architecture for neural network processing in a learned transform domain

An example method of training a neural network includes defining hardware building blocks (HBBs), neuron equivalents (NEQs), and conversion procedures from NEQs to HBBs; defining the neural network using the NEQs in a machine learning framework; training the neural network on a training platform; and converting the neural network as trained into a netlist of HBBs using the conversion procedures to convert the NEQs in the neural network to the HBBs of the netlist.

Learning neural networks of programmable device blocks directly with backpropagation

A device includes a data processing engine array having a plurality of data processing engines organized in a grid having a plurality of rows and a plurality of columns. Each data processing engine includes a core, a memory module including a memory and a direct memory access engine. Each data processing engine includes a stream switch connected to the core, the direct memory access engine, and the stream switch of one or more adjacent data processing engines. Each memory module includes a first memory interface directly coupled to the core in the same data processing engine and one or more second memory interfaces directly coupled to the core of each of the one or more adjacent data processing engines.

Data processing engine arrangement in a device

A circular buffer architecture includes a memory coupled to a producer circuit and a consumer circuit. The memory is configured to store objects. The memory can include memory banks. The number of the memory banks is less than a number of the objects. The circular buffer can include hardware locks configured to reserve selected ones of the memory banks for use by the producer circuit or the consumer circuit. The circular buffer can include a buffer controller coupled to the memory and configured to track a plurality of positions. The positions can include a consumer bank position, a consumer object position, a producer bank position, and a producer object position. The buffer controller is configured to allocate selected ones of the objects from the memory banks to the producer circuit and to the consumer circuit according to the tracked positions and using the hardware locks.

Circular buffer architecture using local memories with limited resources

Examples herein propose operating redundant ML models which have been trained using a boosting technique that considers hardware faults. The embodiments herein describe performing an evaluation process where the performance of a first ML model is measured in the presence of a hardware fault. The errors introduced by the hardware fault can then be used to train a second ML model. In one embodiment, a second evaluation process is performed where the combined performance of both the first and second trained ML models is measured in the presence of a hardware fault. The resulting errors can then be used when training a third ML model. In this manner, the three trained ML models are trained to be error aware. As a result, during operation, if a hardware fault occurs, the three ML models have better performance relative to three ML models that where not trained to be error aware.

Error aware module redundancy for machine learning

An example a method of optimizing a neural network having a plurality of layers includes: obtaining an architecture constraint for circuitry of an inference platform that implements the neural network; training the neural network on a training platform to generate network parameters and feature maps for the plurality of layers; and constraining the network parameters, the feature maps, or both based on the architecture constraint.

Architecture optimized training of neural networks

A device may include a plurality of data processing engines. Each of the data processing engines may include a memory pool having a plurality of memory banks, a plurality of cores each coupled to the memory pool and configured to access the plurality of memory banks, a memory mapped switch coupled to the memory pool and a memory mapped switch of at least one neighboring data processing engine, and a stream switch coupled to each of the plurality of cores and to a stream switch of the at least one neighboring data processing engine.

An integrated circuit (IC) can include a decomposer data mover circuit configured to read sub-arrays from array data stored in a source memory; generate metadata headers for the sub-arrays, wherein each metadata header includes location information indicating location of a corresponding sub-array within the array data; create data tiles, wherein each data tile includes a sub-array and a corresponding metadata header; and output the data tiles to compute circuitry within the IC. The IC can include a composer data mover circuit configured to receive processed versions of the data tiles from the compute circuitry; extract valid data regions from the processed versions of the data tiles; and write the valid data regions to a destination memory based on the location information from the metadata headers of the processed versions of the data tiles.

Data mover circuitry for N-dimensional data in an integrated circuit

Examples herein describe techniques for communicating between data processing engines in an array of data processing engines. In one embodiment, the array is a 2D array where each of the DPEs includes one or more cores. In addition to the cores, the data processing engines can include a memory module (with memory banks for storing data) and an interconnect which provides connectivity between the engines. To transmit processed data, a data processing engine identifies a destination processing engine in the array. Once identified, the data processing engine can transmit the processed data using a reserved point-to-point communication path in the interconnect that couples the source and destination data processing engines.

Streaming interconnect architecture for data processing engine array

A device may include a plurality of data processing engines. Each of the data processing engines may include a core and a memory module. The plurality of data processing engines may be organized in a plurality of rows. Each core may be configured to communicate with other neighboring data processing engines of the plurality of data processing engines by shared access to the memory modules of the neighboring data processing engines.

A device may include a plurality of data processing engines. Each data processing engine may include a core and a memory module. Each core may be configured to access the memory module in the same data processing engine and a memory module within at least one other data processing engine of the plurality of data processing engines.

Device with data processing engine array

The coherent accelerator processor interface (CAPI) provides a high-performance when using heterogeneous compute architectures, but CAPI is not compatible with the advanced extensible interface (AXI) which is used by many accelerators. The examples herein describe an AXI-CAPI adapter (e.g., a hardware architecture) that converts AXI signals to CAPI signals and vice versus. In one example, the AXI-CAPI adapter includes four modules: a low-level shim, a high-level shim, an AXI full module, and an AXI Lite module which are organized in a hierarchy of hardware elements. Each of the modules outputs can output a different version of the AXI signals using the hierarchical structure.

AXI-CAPI adapter

In an example, an integrated circuit (IC) includes a receive circuit, a transmit circuit, and a control circuit. The receive circuit includes a receive data path and a receive control interface, the receive data path coupled to store received transmission control protocol (TCP) data for a plurality of TCP sessions in a respective plurality of receive buffers in an external memory circuit external to the IC. The transmit circuit includes a transmit data path and a transmit control interface, the transmit data path coupled to read TCP data to be transmitted for the plurality of TCP sessions from a respective plurality of transmit buffers in the external memory circuit. The control circuit is coupled to the receive control interface and the transmit control interface, the control circuit configured to maintain data structures to maintain TCP state information for the plurality of TCP sessions.

Data-flow architecture for a TCP offload engine

A circuit for controlling the operation of a memory system having different types of memory is described. The circuit comprises a first memory having a first type of memory element and having a first access time; a second memory having a second type of memory element and having a second access time, wherein the second type of memory element is different than the first type of memory element; a memory control circuit enabling access to the first memory and the second memory; a delay buffer coupled to the second memory to compensate for a difference in the first access time and the second access time; and a circuit for merging outputs of the first memory and delayed outputs of the second memory to generate ordered output data. A method of controlling the operation of a memory system is also disclosed.

Circuits for and methods of controlling the operation of a hybrid memory system

A method of processing data in an integrated circuit is described. The method comprises establishing a pipeline of processing blocks, wherein each processing block has a different function; coupling a data packet having data and meta-data to an input of the pipeline of processing blocks; and processing the data of the data packet using predetermined processing blocks based upon the meta-data. A device for processing data in an integrated circuit is also described.

Growing community of inventors

Sunnyvale, CA, United States of America

Kornelis Antonius Vissers