Google Inc.

Nvidia Corporation

Nuro Corp.

Albert Meixner

Jason Rupert Redgrave

Ofer Shacham

Daniel Finchelstein

Qiuling Zhu

William R Mark

David Patterson

Hyunchul Park

Neeti Desai

Donald C Stark

Edward Y Chang

Artem Vasilyev

Dilan Manatunga

Manu Gulati

Timothy James Knight

Benjamin K Dodge

Shahriar Rabii

Keith Robert Pflederer

Allan Douglas Knies

Vinod Chamarty

Xiaoyu Ma

Hongil Yoon

Don Stark

Rahul Jagdish Thakur

Ji Kim

Weiping Liao

Masumi Reynders

Fabrizio Basso

Michelle Tomasko

Eric Brian Lum

Szepo Robert Hung

Yury Y Uralsky

Rouslan Lyubomirov Dimitrov

Patrick James Neill

Edward T Chang

William Paul Maddern

Ignacio Llamas Ubieto

Yichao (Roger) Shen

Alexandr Bakhturin

Chenjie Luo

Ian Zhou

Hubert Hua Kian Teo

Young Jean (Albert) Han

Zhan Xiong Chin

Dustin Michael DeWeese

Daniel Frederic Finchelstein

Dustin Michael De Weese

Daniel Fredric Finchelstein

A method is described that includes executing a convolutional neural network layer on an image processor having an array of execution lanes and a two-dimensional shift register. The two-dimensional shift register provides local respective register space for the execution lanes. The executing of the convolutional neural network includes loading a plane of image data of a three-dimensional block of image data into the two-dimensional shift register. The executing of the convolutional neural network also includes performing a two-dimensional convolution of the plane of image data with an array of coefficient values by sequentially: concurrently multiplying within the execution lanes respective pixel and coefficient values to produce an array of partial products; concurrently summing within the execution lanes the partial products with respective accumulations of partial products being kept within the two dimensional register for different stencils within the image data; and, effecting alignment of values for the two-dimensional convolution within the execution lanes by shifting content within the two-dimensional shift register array.

Convolutional neural network on programmable two dimensional image processor

According to one aspect, a method includes identifying a first signal, which carries or otherwise includes data, to be provided to a system by a vehicle. The method includes determining a first amount of bandwidth, the first amount of bandwidth being associated with the signal, determining a second amount of bandwidth, the second amount of bandwidth being an available bandwidth associated with a plurality of modems included on the vehicle to enable the vehicle to communicate with the system, and dividing the first signal into a first plurality of portions based at least on the first amount of bandwidth and the second amount of bandwidth. The first plurality of portions is provided to the plurality of modems for transmission from the vehicle to the system, and is transmitted to the system using the plurality of modems and a plurality of channels associated with the plurality of modems.

Methods and apparatus for providing redundant networking capabilities for teleoperations

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for a system-level cache to allocate cache resources by a way-partitioning process. One of the methods includes maintaining a mapping between partitions and priority levels and allocating primary ways to respective enabled partitions in an order corresponding to the respective priority levels assigned to the enabled partitions.

Way partitioning for a system-level cache

An image processor is described. The image processor includes a two dimensional shift register array that couples certain ones of its array locations to support execution of a shift instruction. The shift instruction is to include mask information. The mask information is to specify which of the array locations are to be written to with information being shifted. The two dimensional shift register array includes masking logic circuitry to write the information being shifted into specified ones of the array locations in accordance with the mask information.

Two dimensional masked shift instruction

Techniques for image quality enhancement for autonomous vehicle remote operations are disclosed herein. An image processing system of an autonomous vehicle can obtain images captured by at least two different cameras and stitch the images together to create a combined image. The image processing system can apply region blurring to a portion of the combined image to create an enhanced combined image, e.g., to blur regions/objects determined to be less import (or unimportant) for the remote operations. The image processing system can encode pixel areas of the enhanced combined image using a corresponding quality setting for respective pixel areas to create encoded image files, e.g., based on complexity levels of the respective pixel areas. The image processing system can transmit the encoded image files to a remote operations system associated with the autonomous vehicle for remote operations support.

Image quality enhancement for autonomous vehicle remote operations

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for supporting large lookup tables on an image processor. One of the methods includes receiving an input kernel program for an image processor having a two-dimensional array of execution lanes, a shift-register array, and a plurality of memory banks. If the kernel program has an instruction that reads a lookup table value for a lookup table partitioned across the plurality of memory banks, the instruction in the kernel program are replaced with a sequence of instructions that, when executed by an execution lane, causes the execution lane to read a first value from a local memory bank and a second value from the local memory bank on behalf of another execution lane belonging to a different group of execution lanes.

Large lookup tables for an image processor

An image processor unit is described. The image processor unit includes a plurality of inputs to receive at least one input image. The image processor unit includes a plurality of outputs to provide at least one output image. The image processor unit includes a network coupled to the plurality of inputs and the plurality of outputs. The network is to couple at least one of the inputs to at least one of the outputs. The image processor unit includes an image processor circuit coupled to the network. The network to route an input image that is received at one of the inputs to the image processor circuit. The image processor circuit is to execute image signal processing program code to generate a processed output image from the input image. The network is to route the processed output image to at least one of the outputs.

Configurable and programmable image processor unit

A method is described that includes, on an image processor having a two dimensional execution lane array and a two dimensional shift register array, repeatedly shifting first content of multiple rows or columns of the two dimensional shift register array and repeatedly executing at least one instruction between shifts that operates on the shifted first content and/or second content that is resident in respective locations of the two dimensional shift register array that the shifted first content has been shifted into.

Block operations for an image processor having a two-dimensional execution lane array and a two-dimensional shift register

An apparatus is described that include a line buffer unit composed of a plurality of a line buffer interface units. Each line buffer interface unit is to handle one or more requests by a respective producer to store a respective line group in a memory and handle one or more requests by a respective consumer to fetch and provide the respective line group from memory. The line buffer unit has programmable storage space whose information establishes line group size so that different line group sizes for different image sizes are storable in memory.

Line buffer unit for image processor

A method is described that includes translating higher level program code including higher level instructions having an instruction format that identifies pixels to be accessed from a memory with first and second coordinates from an orthogonal coordinate system into lower level instructions that target a hardware architecture having an array of execution lanes and a shift register array structure that is able to shift data along two different axis. The translating includes replacing the higher level instructions having the instruction format with lower level shift instructions that shift data within the shift register array structure.

Compiler for translating between a virtual image processor instruction set architecture (ISA) and target hardware having a two-dimensional shift array structure

An apparatus is described. The apparatus includes an execution lane array coupled to a two dimensional shift register array structure. Locations in the execution lane array are coupled to same locations in the two-dimensional shift register array structure such that different execution lanes have different dedicated registers.

Two dimensional shift array for image processor

An apparatus that includes a program controller to fetch and issue instructions is described. The apparatus includes an execution lane having at least one execution unit to execute the instructions. The execution lane is part of an execution lane array that is coupled to a two dimensional shift register array structure, wherein, execution lane s of the execution lane array are located at respective array locations and are coupled to dedicated registers at same respective array locations in the two-dimensional shift register array.

Energy efficient processor core architecture for image processor

A sheet generator circuit is described. The sheet generator includes electronic circuitry to receive a line group of image data including multiple rows of data from a frame of image data. The multiple rows are sufficient in number to encompass multiple neighboring overlapping stencils. The electronic circuitry is to parse the line group into a smaller sized sheet. The electronic circuitry is to load the sheet into a data computation unit having a two dimensional shift array structure coupled to an array of processors.

Sheet generator for image processor

A method is described. The method includes calculating data transfer metrics for kernel-to-kernel connections of a program having a plurality of kernels that is to execute on an image processor. The image processor includes a plurality of processing cores and a network connecting the plurality of processing cores. Each of the kernel-to-kernel connections include a producing kernel that is to execute on one of the processing cores and a consuming kernel that is to execute on another one of the processing cores. The consuming kernel is to operate on data generated by the producing kernel. The method also includes assigning kernels of the plurality of kernels to respective ones of the processing cores based on the calculated data transfer metrics.

Configuration of application software on multi-core image processor

A method is described. The method includes constructing an image processing software data flow in which a buffer stores and forwards image data being transferred from a producing kernel to one or more consuming kernels. The method also includes recognizing that the buffer has insufficient resources to store and forward the image data. The method also includes modifying the image processing software data flow to include multiple buffers that store and forward the image data during the transfer of the image data from the producing kernel to the one or more consuming kernels.

Program code transformations to improve image processor runtime efficiency

Growing community of inventors

Mountain View, CA, United States of America

Albert Meixner