Apple Inc.

Magnum Semiconductor, Inc.

Calos Fund Limited Liability Company

Mark P Rygh

Jim Chen Chou

Guy Cote

Joseph J Cheng

Athanasios Leontaris

Timothy John Millet

Guy Côté

John L Melanson

Peter F Holland

Joseph P Bratt

Sorin Constantin Cismas

Brucek Kurdo Khailany

Marc A Schaub

Craig M Okruhlica

Udo Uebel

Jae Y Park

Jim Jian Lin

Miles Simpson

Liviu R Morogan

Gaurav S Gulati

Dan Bell

Guy Cõté

A video encoding system encodes source image data corresponding with an image includes a low resolution pipeline that receives the source image data corresponding with a first coding block in the image. The low resolution pipeline includes a low resolution motion estimation block programmed to generate a first downscaled coding block by downscaling resolution of the source image data corresponding with the first coding block. The first downscaled coding block comprises a first downscaled prediction block corresponding with a first prediction block in the first coding block. The low resolution pipeline may also perform several low resolution motion estimation searches to generate motion vector candidates. The video encoding system also includes a main pipeline that receives the source image data and determines encoding parameters to be used to encode the first coding block based at least partially on the motion vector candidates.

Systems and methods for low resolution motion estimation searches

The present disclosure relates to systems and methods of multi-processing core processing of image frames during image encoding. The multiple processing cores may be connected via dedicated interfaces and transfer neighbor data between the processing cores to enable parallel processing of frame data. The multiple processing cores may each process quad-rows of image data for a single frame in parallel to reduce memory usage and mitigate latency in video encoding.

Systems and methods for multi-core image encoding

Systems and methods for improving operational efficiency of a video encoding system used to encode image data are provided. In embodiments, the video encoding system includes a low resolution pipeline that includes a low resolution motion estimation block, which generates downscaled image data by reducing resolution of the image data and performs a motion estimation search using the downscaled image data and previously downscaled image data. The video encoding system also includes a main encoding pipeline in parallel with the low resolution pipeline that includes a motion estimation block, which determines a global motion vector based on data from the low resolution motion estimation block. The main encoding pipeline may utilize the global motion vector in determining a candidate inter prediction mode.

Global motion vector video encoding systems and methods

Systems and methods for improving operational efficiency of a video encoding system used to encode image data are provided. In embodiments, the video encoding system includes a low resolution pipeline that includes a low resolution motion estimation block, which generates downscaled image data by reducing resolution of the image data, performs a motion estimation search to determine motion vector candidates that are each indicative of location of a downscaled reference sample, and prunes one or more of the motion vector candidates. The video encoding system also includes a main pipeline that determines encoding parameters to be used to encode image data based at least partially the remaining motion vector candidates.

Motion vector candidate pruning systems and methods

Block processing pipeline methods and apparatus in which. motion estimation is performed at a stage of a motion estimation module for a current block with respect to a reference frame at one or more partition sizes to determine candidate motion vectors. The candidate motion vectors may be passed to a next stage for refinement. Motion estimation may then be performed at the next stage to refine the motion vectors. In performing motion estimation at this stage, the input motion vectors of at least one partition size received from the previous stage may be used as candidate motion vectors in searches for at least one other partition size.

Motion estimation in block processing pipelines

System and method for improving operational efficiency of a video encoding pipeline used to encode image data. The video encoding pipeline includes a mode decision block, which selects a first inter-frame prediction mode used to prediction encode a first prediction unit, and a motion estimation block, which receives the first inter-frame prediction mode as feedback from the mode decision block when processing a second prediction unit; determines an initial candidate inter-frame prediction mode of the second prediction unit based at least in part on the first inter-frame prediction mode; and determines a final candidate inter-frame prediction mode of the second prediction unit by performing a first motion estimation search based at least in part on the initial candidate inter-frame prediction mode. The mode decision block determines a rate-distortion cost associated with the final candidate inter-frame prediction mode and a prediction mode used to prediction encode the second prediction unit based at least in part on the rate-distortion cost.

Predictor candidates for motion estimation search systems and methods

System and method for improving operational efficiency of a video encoding pipeline used to encode image data. The video encoding pipeline includes a motion estimation setup block, which dynamically adjusts a setup configuration of the motion estimation block based at least in part on operational parameters of the video encoding pipeline and select an initial candidate inter-frame prediction mode based at least on the setup configuration, a full-pel motion estimation block, which determines an intermediate candidate inter-frame prediction mode by performing a motion estimation search based on the initial candidate inter-frame prediction mode, a sub-pel motion estimation block, which determines a final candidate inter-frame prediction by performing a motion estimation search based on the intermediate candidate inter-frame prediction mode, and a mode decision block, which determines a rate-distortion cost associated with the final candidate inter-frame prediction mode and determines a prediction mode used to prediction encoding the image data.

Configurable motion estimation search systems and methods

System and method for improving operational efficiency of a video encoding pipeline used to encode image data. In embodiments, the video encoding pipeline includes a low resolution pipeline that includes a low resolution motion estimation block, which generates downscaled image data by reducing resolution of the image data and determines a low resolution inter-frame prediction mode by performing a motion estimation search using the downscaled image data and previously downscaled image data. The video encoding pipeline also includes a main pipeline in parallel with the low resolution pipeline that includes a motion estimation block, which determines a candidate inter-frame prediction mode based at least in part on the low resolution inter-frame prediction mode, and a mode decision block, which determines a first rate-distortion cost associated with the candidate inter-frame prediction mode and determines prediction mode used to prediction encode the image data based at least in part on the first rate-distortion cost.

Memory-to-memory low resolution motion estimation systems and methods

Block processing pipeline methods and apparatus in which reference data are stored to a memory according to tile formats to reduce memory accesses when fetching the data from the memory. When the pipeline stores reference data from a current frame being processed to memory as a reference frame, the reference samples are stored in macroblock sequential order. Each macroblock sample set is stored as a tile. Reference data may be stored in tile formats for luma and chroma. Chroma reference data may be stored in tile formats for chroma 4:2:0, 4:2:2, and/or 4:4:4 formats. A stage of the pipeline may write luma and chroma reference data for macroblocks to memory according to one or more of the macroblock tile formats in a modified knight's order. The stage may delay writing the reference data from the macroblocks until the macroblocks have been fully processed by the pipeline.

Data storage and access in block processing pipelines

In an embodiment, a system includes a display processing unit configured to process a video sequence for a target display. In some embodiments, the display processing unit is configured to composite the frames from frames of the video sequence and one or more other image sources. The display processing unit may be configured to write the processed/composited frames to memory, and may also be configured to generate statistics over the frame data, where the generated statistics are usable to encode the frame in a video encoder. The display processing unit may be configured to write the generated statistics to memory, and the video encoder may be configured to read the statistics and the frames. The video encoder may be configured to encode the frame responsive to the statistics.

Display pipe statistics calculation for video encoder

In the video encoders described herein, blocks of pixels from a video frame may be encoded (e.g., using CAVLC encoding) in a block processing pipeline using wavefront ordering (e.g., in knight's order). Each of the encoded blocks may be written to a particular one of multiple DMA buffers such that the encoded blocks written to each of the buffers represent consecutive blocks of the video frame in scan order. A transcode pipeline may operate in parallel with (or at least overlapping) the operation of the block processing pipeline. The transcode pipeline may read encoded blocks from the buffers in scan order and merge them into a single bit stream (in scan order). A transcoder core of the transcode pipeline may decode the encoded blocks and encode them using a different encoding process (e.g., CABAC). In some cases, the transcoder may be bypassed.

Wavefront encoding with parallel bit stream encoding

Block processing pipeline methods and apparatus in which pixel data from a reference frame is prefetched into a search window memory. The search window may include two or more overlapping regions of pixels from the reference frame corresponding to blocks from the rows in the input frame that are currently being processed in the pipeline. Thus, the pipeline may process blocks from multiple rows of an input frame using one set of pixel data from a reference frame that is stored in a shared search window memory. The search window may be advanced by one column of blocks by initiating a prefetch for a next column of reference data from a memory. The pipeline may also include a reference data cache that may be used to cache a portion of a reference frame and from which at least a portion of a prefetch for the search window may be satisfied.

Reference frame data prefetching in block processing pipelines

Memory latency tolerance methods and apparatus for maintaining an overall level of performance in block processing pipelines that prefetch reference data into a search window. In a general memory latency tolerance method, search window processing in the pipeline may be monitored. If status of search window processing changes in a way that affects pipeline throughput, then pipeline processing may be modified. The modification may be performed according to no stall methods, stall recovery methods, and/or stall prevention methods. In no stall methods, a block may be processed using the data present in the search window without waiting for the missing reference data. In stall recovery methods, the pipeline is allowed to stall, and processing is modified for subsequent blocks to speed up the pipeline and catch up in throughput. In stall prevention methods, processing is adjusted in advance of the pipeline encountering a stall condition.

Memory latency tolerance in block processing pipelines

A knight's order processing method for block processing pipelines in which the next block input to the pipeline is taken from the row below and one or more columns to the left in the frame. The knight's order method may provide spacing between adjacent blocks in the pipeline to facilitate feedback of data from a downstream stage to an upstream stage. The rows of blocks in the input frame may be divided into sets of rows that constrain the knight's order method to maintain locality of neighbor block data. Invalid blocks may be input to the pipeline at the left of the first set of rows and at the right of the last set of rows, and the sets of rows may be treated as if they are horizontally arranged rather than vertically arranged, to maintain continuity of the knight's order algorithm.

Growing community of inventors

Union City, CA, United States of America

Mark P Rygh