Meta Platforms, Inc.

Nvidia Corporation

Qualcomm Incorporated

Harikrishna Madadi Reddy

Yunqing Chen

Gaurang Chaudhari

Ignatius B Tjandrasuwita

Iole Moccagatta

Junqiang Lan

Visalakshi Vaduganathan

Baheerathan Anandharengan

Xianliang Zha

Zhao Wang

Srikanth Alaparthi

Sujith Srinivasan

Guogang Hua

Chung-Fu Lin

Christian Markus Petersen

Manindra Parhy

Hariharan Ganesh Lalgudi

Xing Cindy Chen

Shiyan Pan

Prahlad Rao Venkatapuram

Deepa Palamadai Sundar

Arvind Srinivasan

Christopher L Mills

Ravi Prasad Bulusu

Atul Garg

Wei Sun

Vinayak Jayaram Pore

Lefan Zhong

Ashish Kumar Mishra

Naader Hasani

Derek E Pappas

Leonardo Vainsencher

Shankar Moni

Himadri Choudhury

Matthew Raymond Longnecker

Liang Cheng

Pankaj Kansal

Soumenkumar Dey

Shashank Garg

Ram Prabhakar

Yiu Cheong Ho

Cheng-Chiang Chen

Manish Jatashanker Pandey

Tushar Singhal

In some embodiments, a computer-implemented method includes receiving, at a security agent of a host central processing unit (CPU), accelerator firmware from flash memory; determining, at the security agent, whether the accelerator firmware includes a critical accelerator firmware component or a non-critical accelerator firmware component; authenticating, at the security agent, the critical accelerator firmware component instantaneously upon a determination that the accelerator firmware is the critical accelerator firmware component, wherein authenticating the critical accelerator firmware component yields an authenticated critical accelerator firmware component; and providing the authenticated critical accelerator firmware component to an accelerator via a sideband bus for execution at the accelerator.

Integrated chiplet-based central processing units with accelerators for system security

In some embodiments, a system-on-chip, includes a central processing unit (CPU); an accelerator coupled to the CPU via a first die-to-die interconnect; and uniform memory coupled to the CPU via a second die-to-die interconnect. In some embodiments, in order to prevent use of accelerator memory for processing operations by the accelerator, the accelerator utilizes a uniform memory access tunneling system located in the accelerator to tunnel a high-level interconnect protocol associated with the second die-to-die interconnect to a die-to-die interconnect protocol associated with the first die-to-die interconnect, the uniform memory access tunneling system being configured to allow access to the uniform memory using a shared address space.

Integrated chiplet-based central processing units with accelerators

Techniques to optimize memory reads when computing a video quality metric are disclosed. In some embodiments, an application-specific integrated circuit for computing video quality metrics includes a set of caches configured to store neighbor pixel data for edge width searches of pixels comprising a frame of a video being analyzed for a video quality metric and a kernel configured to receive corresponding neighbor pixel data for pixels comprising a current processing block of the frame from a subset of the set of caches and simultaneously perform edge width searches for pixels comprising the current processing block to determine corresponding pixel edge width values used for computing the video quality metric.

Optimizing memory reads when computing video quality metrics

A system includes a first integrated circuit package including a first group of one or more artificial intelligence processing units and a first chip-to-chip interconnect communication unit and a second integrated circuit package including a second group of one or more artificial intelligence processing units and a second chip-to-chip interconnect communication unit. The system also includes an interconnect between the first integrated circuit package and the second integrated circuit package, wherein the first chip-to-chip interconnect communication unit and the second chip-to-chip interconnect communication unit manage ethernet-based communication via the interconnect using a layered communication architecture supporting a credit-based data flow control and a retransmission data flow control.

Chip-to-chip interconnect with a layered communication architecture

A scalable hardware accelerator configured to compute video quality metrics is disclosed. In some embodiments, an accelerator for video quality metrics comprises an application-specific integrated circuit that includes a buffer memory configured to store at least a portion of a reference frame of a video and at least a corresponding portion of a distorted frame of a transcoded version of the video and that includes a processing unit configured to receive data from the buffer memory and compute a perception-based video quality metric for the distorted frame with respect to the reference frame.

Scalable accelerator architecture for computing video quality metrics

An example disclosed system may include a hardware video encoding pipeline (HVEP), the HVEP that includes a prediction module that generates, using a primary prediction mode, a primary encode of a portion of a video stream, and using a secondary prediction mode, a secondary encode of the portion of the video stream. The HVEP may also include (1) a rate-distortion optimization (RDO) module that determines a primary cost associated with the primary encode and a secondary cost associated with the secondary encode, (2) a quality metric (QM) module that determines a primary QM associated with the primary encode and a secondary QM associated with the secondary encode, and (3) a decision module that selects, for an encoding of the video stream, at least one of the primary prediction mode and the secondary prediction mode based on the primary cost, the secondary cost, the primary QM, and the secondary QM.

Systems and methods for improving mode decisions in hardware video encoders

A disclosed system may include a hardware distortion data pipeline that may include (1) a quantization module that generates a quantized data set, (2) an inverse quantization module that generates, from the quantized data set, an inverse quantized data set by executing an inverse quantization of the quantized data set, and (3) an inverse transformation module that generates an inversely transformed data set by executing an inverse transformation of the inverse quantized data set. The system may also include a hardware determination pipeline that determines a distortion metric based on the inversely transformed data set and the residual frame data set, and a hardware token rate pipeline that determines, based on the quantized data set, a token rate for an encoding of the residual frame data set via a video encoding pipeline. Various other methods, systems, and computer-readable media are also disclosed.

Hardware pipelines for rate-distortion optimization (RDO) that support multiple codecs

Techniques for detecting skipped prediction units in an encoder are disclosed. Control information associated with a prediction unit of a video frame is received at a node of a video encoding pipeline used to compress the video frame. A skip decision for the prediction unit is made by analyzing transform units comprising the prediction unit. The skip decision indicates whether or not the prediction unit comprises a skipped prediction unit during encoding. The skip decision for the prediction unit is provided to a downstream node of the video encoding pipeline.

Prediction unit skip detection in encoder

An example system may include a primary mode decision module, included in a hardware video encoding pipeline, that (1) receives video data for encoding in accordance with a video encoding standard, and (2) identifies, from an initial set of prediction modes supported by the video encoding standard, a primary set of prediction modes for encoding the video data in accordance with the video encoding standard. The example system may also include a secondary mode decision module that (1) determines, for each prediction mode included in the primary set of prediction modes and based on the video data, a cost associated with the prediction mode, and (2) selects, from the primary set of prediction modes and based on the determined costs associated with the prediction modes included in the primary set of prediction modes, a prediction mode for encoding of the video data by the hardware video encoding pipeline.

Systems and methods for dynamic early termination of mode decision in hardware video encoders

A count circuit for symbol statistics is disclosed that is configured to read from an address of a buffer memory a count value stored at the address in response to receiving a first of a plurality of input values comprising the address, serially increment the count value for each of the received plurality of input values comprising the address, and write an incremented count value at the address of the buffer memory after a last of the plurality of input values comprising the address has been counted. Reading from the buffer memory is disabled for all but the first of the plurality of input values comprising the address. Writing to the buffer memory is disabled for all but the last of the plurality of input values comprising the address.

Count circuit for symbol statistics

A disclosed computer-implemented method may include (1) selecting, from a video stream, a reference frame and a current frame, (2) collecting a reference histogram of the reference frame and a current histogram of the current frame, and (3) generating a smoothed reference histogram by applying a smoothing function to at least a portion of the reference histogram. In some examples, the computer-implemented method may also include (1) determining a similarity metric between the smoothed reference histogram and the current histogram and, (2) when the similarity metric is greater than a threshold value, applying weighted prediction during a motion estimation portion of an encoding of the video stream. Various other methods, systems, and computer-readable media are also disclosed.

Systems and methods for histogram-based weighted prediction in video encoding

A mode decision component is configured to determine the costs of different modes for a selected partition of a frame of a video using an estimated compression coding data, which is calculated prior to a corresponding actual compression coding data being calculated based on another partition immediately prior to the selected partition in a partition processing order. The estimated compression coding data is determined based on a previously calculated compression coding data calculated based on a completed partition prior to the selected partition in the partition processing order. The mode decision component is configured to use the determined costs to select one of the modes. An encoder component is configured to use the selected mode to encode the selected partition by using the corresponding actual compression coding data calculated based on the another partition immediately prior to the selected partition in the partition processing order.

Architecture to adapt cumulative distribution functions for mode decision in video encoding

A system comprises a memory storage configured to store at least a portion of a frame of a video and a hardware motion estimation search processing unit configured to perform at least a portion of a motion estimation search for the video for a plurality of different block sizes. The hardware motion estimation search processing unit is configured to perform the motion estimation search using a plurality of source sub-blocks of a first block size to determine a first type of comparison evaluation values for the first block size. A combination of values included in the first type of comparison evaluation values is utilized to determine at least one second type of comparison evaluation values for a second block size, wherein the second block size is larger than the first block size.

Unified search window to support multiple video encoding standards

A method for sharing the motion estimation and mode decision results and decisions of one codec with another codec is disclosed. A video is received to be transcoded into a plurality of different output encodings of a plurality of different codecs. Each codec has a different video encoding format. A shared motion estimation and a shared mode decision processing of the video are performed. One or more results of the shared mode decision processing shared across the plurality of different codecs are used to encode the video into the plurality of different output encodings of the plurality of different codecs.

Cross-codec encoding optimizations for video transcoding

A system for calculating token rates for video encoding includes a plurality of different probability lookup tables implemented in hardware, wherein each of the probability lookup tables specifically corresponds to a different prediction mode of a video codec. The system includes an application-specific integrated circuit compute unit. For each candidate prediction mode among the different prediction modes, the application-specific integrated circuit is configured to determine a rate distortion cost (RD Cost) for a video. The application-specific integrated circuit is configured to select one of the plurality of different probability lookup tables that corresponds to the candidate prediction mode and use the selected one of the plurality of different probability lookup tables to calculate a corresponding token rate for the candidate prediction mode. The application-specific integrated circuit is configured to encode the video using a selected one of the different prediction modes determined based on the determined rate distortion costs.

Hardware optimization of rate calculation in rate distortion optimization for video encoding

Growing community of inventors

San Jose, CA, United States of America

Harikrishna Madadi Reddy