Meta Platforms, Inc.

Qualcomm Incorporated

Yunqing Chen

Harikrishna Madadi Reddy

Gaurang Chaudhari

Baheerathan Anandharengan

Zhao Wang

Srikanth Alaparthi

Junqiang Lan

Shiyan Pan

Christian Markus Petersen

Xianliang Zha

Kameswara Kishore Sriadibhatla

Prahlad Rao Venkatapuram

Adrian Stafford Lewis

Ashish Kumar Mishra

Hariharan Ganesh Lalgudi

Cheng-Chiang Chen

Prasanth Jampani

Chien Cheng Liu

Tushar Singhal

Anil Muthiraparampil Sunil

In some embodiments, a computer-implemented method includes receiving, at a security agent of a host central processing unit (CPU), accelerator firmware from flash memory; determining, at the security agent, whether the accelerator firmware includes a critical accelerator firmware component or a non-critical accelerator firmware component; authenticating, at the security agent, the critical accelerator firmware component instantaneously upon a determination that the accelerator firmware is the critical accelerator firmware component, wherein authenticating the critical accelerator firmware component yields an authenticated critical accelerator firmware component; and providing the authenticated critical accelerator firmware component to an accelerator via a sideband bus for execution at the accelerator.

Integrated chiplet-based central processing units with accelerators for system security

In some embodiments, a system-on-chip, includes a central processing unit (CPU); an accelerator coupled to the CPU via a first die-to-die interconnect; and uniform memory coupled to the CPU via a second die-to-die interconnect. In some embodiments, in order to prevent use of accelerator memory for processing operations by the accelerator, the accelerator utilizes a uniform memory access tunneling system located in the accelerator to tunnel a high-level interconnect protocol associated with the second die-to-die interconnect to a die-to-die interconnect protocol associated with the first die-to-die interconnect, the uniform memory access tunneling system being configured to allow access to the uniform memory using a shared address space.

Integrated chiplet-based central processing units with accelerators

According to examples, a system for implementing image modification functions via use of variable scanning orders is described. The system may include a processor and a memory storing instructions. The processor, when executing the instructions, may cause the system to partition an image into a plurality of image blocks, identify one or more image blocks of the plurality of image blocks associated with a region of interest (ROI), and scan the one or more image blocks in an image modification order. The processor, when executing the instructions, may then arrange the one or more image blocks according to the image modification order to form a modified image including the region of interest (ROI) and crop the region of interest (ROI) in the modified image to form a new image.

Implementing image modification functions using variable scanning orders

A video encoder is disclosed. The video encoder comprises an integer level motion estimation hardware component configured to determine candidate integer level motion vectors for a video being encoded. The video encoder comprises a fractional motion estimation hardware component configured to receive the candidate integer level motion vectors from the integer level motion estimation hardware component and refine the candidate integer level motion vectors into candidate sub-pixel level motion vectors. The fractional motion estimation hardware component includes parallel pipelines configured to process coding units of a frame of the video in parallel across the parallel pipelines.

Fractional motion estimation engine with parallel code unit pipelines

An example disclosed system may include a hardware video encoding pipeline (HVEP), the HVEP that includes a prediction module that generates, using a primary prediction mode, a primary encode of a portion of a video stream, and using a secondary prediction mode, a secondary encode of the portion of the video stream. The HVEP may also include (1) a rate-distortion optimization (RDO) module that determines a primary cost associated with the primary encode and a secondary cost associated with the secondary encode, (2) a quality metric (QM) module that determines a primary QM associated with the primary encode and a secondary QM associated with the secondary encode, and (3) a decision module that selects, for an encoding of the video stream, at least one of the primary prediction mode and the secondary prediction mode based on the primary cost, the secondary cost, the primary QM, and the secondary QM.

Systems and methods for improving mode decisions in hardware video encoders

A disclosed system may include a hardware distortion data pipeline that may include (1) a quantization module that generates a quantized data set, (2) an inverse quantization module that generates, from the quantized data set, an inverse quantized data set by executing an inverse quantization of the quantized data set, and (3) an inverse transformation module that generates an inversely transformed data set by executing an inverse transformation of the inverse quantized data set. The system may also include a hardware determination pipeline that determines a distortion metric based on the inversely transformed data set and the residual frame data set, and a hardware token rate pipeline that determines, based on the quantized data set, a token rate for an encoding of the residual frame data set via a video encoding pipeline. Various other methods, systems, and computer-readable media are also disclosed.

Hardware pipelines for rate-distortion optimization (RDO) that support multiple codecs

Techniques for detecting skipped prediction units in an encoder are disclosed. Control information associated with a prediction unit of a video frame is received at a node of a video encoding pipeline used to compress the video frame. A skip decision for the prediction unit is made by analyzing transform units comprising the prediction unit. The skip decision indicates whether or not the prediction unit comprises a skipped prediction unit during encoding. The skip decision for the prediction unit is provided to a downstream node of the video encoding pipeline.

Prediction unit skip detection in encoder

A count circuit for symbol statistics is disclosed that is configured to read from an address of a buffer memory a count value stored at the address in response to receiving a first of a plurality of input values comprising the address, serially increment the count value for each of the received plurality of input values comprising the address, and write an incremented count value at the address of the buffer memory after a last of the plurality of input values comprising the address has been counted. Reading from the buffer memory is disabled for all but the first of the plurality of input values comprising the address. Writing to the buffer memory is disabled for all but the last of the plurality of input values comprising the address.

Count circuit for symbol statistics

The controller is configured to receive a request that causes transferring of a reference pixel block of a video, wherein the video is being encoded using the reference pixel block. The controller is configured to determine whether the reference pixel block has at least a portion that is outside a frame of the video. In response to the determination of the reference pixel block having at least a portion outside the frame of the video, the controller is configured to cause a portion of the reference pixel block of the video inside the frame of the video to be fetched from a memory storage and stored in a cache storage, pad a remaining missing portion of the reference pixel block of the video outside the frame of the video with padding pixel data to form the reference pixel block, and transfer the reference pixel block in response to the request.

Supporting multiple partition sizes using a unified pixel input data interface for fetching reference pixels in video encoders

A video encoder comprises an application-specific integrated circuit (ASIC) video encoding unit configured to receive via an interface in a processing order, quantized transform coefficients for a transform unit of a frame of a video being encoded. The ASIC unit is configured to analyze at least a portion of the quantized transform coefficients in the processing order to identify one or more end-of-block candidate positions in the processing order. The ASIC unit is configured to translate the one or more end-of-block candidate positions for the at least a portion of the quantized transform coefficients to one or more scan order versions of the one or more end-of-block candidate positions. The ASIC unit is configured to determine a true end-of-block position for the quantized transform coefficients using the one or more scan order versions of the one or more end-of-block candidate positions.

End-of-block detection of quantized transform coefficient matrices in video encoding

A mode decision component is configured to determine the costs of different modes for a selected partition of a frame of a video using an estimated compression coding data, which is calculated prior to a corresponding actual compression coding data being calculated based on another partition immediately prior to the selected partition in a partition processing order. The estimated compression coding data is determined based on a previously calculated compression coding data calculated based on a completed partition prior to the selected partition in the partition processing order. The mode decision component is configured to use the determined costs to select one of the modes. An encoder component is configured to use the selected mode to encode the selected partition by using the corresponding actual compression coding data calculated based on the another partition immediately prior to the selected partition in the partition processing order.

Architecture to adapt cumulative distribution functions for mode decision in video encoding

A system for calculating token rates for video encoding includes a plurality of different probability lookup tables implemented in hardware, wherein each of the probability lookup tables specifically corresponds to a different prediction mode of a video codec. The system includes an application-specific integrated circuit compute unit. For each candidate prediction mode among the different prediction modes, the application-specific integrated circuit is configured to determine a rate distortion cost (RD Cost) for a video. The application-specific integrated circuit is configured to select one of the plurality of different probability lookup tables that corresponds to the candidate prediction mode and use the selected one of the plurality of different probability lookup tables to calculate a corresponding token rate for the candidate prediction mode. The application-specific integrated circuit is configured to encode the video using a selected one of the different prediction modes determined based on the determined rate distortion costs.

Hardware optimization of rate calculation in rate distortion optimization for video encoding

A quantized transform coefficient matrix is partitioned into a sequence of partition portions. The coefficients of the matrix are grouped into the sequence of partition portions based on a hardware implemented scan order. Each partition portion is processed in an order of the sequence in a first pass. For each partition portion, a group of coefficients in the partition portion is determined. For each partition portion, a first data rate estimation for the quantized transform coefficient matrix is updated based on at least some coefficients of the group of coefficients in the partition portion and a maximum end-of-block. For each partition portion, an end-of-block estimation of the quantized transform coefficient matrix is updated based on at least some coefficients of the group of coefficients in the partition portion. A first resulting data rate estimation and a true end-of-block of the quantized transform coefficient matrix are determined after the first pass.

Architecture for rate estimation in video coding

Techniques are described for performing transformation on video data. A transform circuit may receive M sample values of the video data from a pre-transform buffer, and process the M sample values with N computation units of the transform circuit to generate intermediate values. Processing the M sample values to generate the intermediate values includes feeding back temporary values from output of one or more of the N computation units to input of one or more of the N computation units. The transform circuit may store a first set of the intermediate values in a transpose buffer, and store a second set of the intermediate values in the pre-transform buffer that are to be later retrieved for storage in the transpose buffer.

Growing community of inventors

Los Altos, CA, United States of America

Yunqing Chen