Apple Inc.

Douglas Scott Price

Stephen Olivier Lemay

Hsi-Jung Wu

Xiaosong Zhou

James O Normile

Chris Yoochang Chung

Yeping Su

Dazhong Zhang

Joe S Abuan

Hyeonkuk Jeong

Ke Zhang

David L Biderman

James David Batson

Roberto Garcia, Jr

Thomas Christopher Jansen

Elizabeth Caroline Cranfill

Ting Chen

Hao Pan

Yan Yang

Roberto Garcia

John Y Su

Albert E Keinath

Zhenheng Li

Jiefu Zhai

Haitao Guo

Berkat S Tung

David Daming Kuo

Andrew Yanowitz

Yao-Chung Lin

Shujie Liu

David A Eldred

Thomas Jansen

David D Kuo

James Oliver Normile

A video streaming method for transitioning between multiple sequences of coded video data may include receiving and decoding transmission units from a first sequence of coded video data. In response to a request to transition to a second sequence of coded video data, the method may determine whether a time to transition to the second sequence of coded video data can be reduced by transitioning to the second sequence of coded video data via an intermediate sequence of coded video data. If the time can be reduced, the method may include receiving at least one transmission unit from an intermediate sequence of coded video data that corresponds to the request to transition, decoding the transmission unit from the intermediate sequence, and transitioning from the first sequence to the second sequence via the decoded transmission unit from the intermediate sequence.

Codec techniquest for fast switching without a synchronization frame

Systems and processes for improved video editing, summarization and navigation based on generation and analysis of metadata are described. The metadata may be content-based (e.g., differences between neighboring frames, exposure data, key frame identification data, motion data, or face detection data) or non-content-based (e.g., exposure, focus, location, time) and used to prioritize and/or classify portions of video. The metadata may be generated at the time of image capture or during post-processing. Prioritization information, such as a score for various portions of the image data may be based on the metadata and/or image data. Classification information such as the type or quality of a scene may be determined based on the metadata and/or image data. The classification and prioritization information may be metadata and may be used to automatically remove undesirable portions of the video, generate suggestions during editing or automatically generate summary video.

Video analysis techniques for improved editing, navigation, and summarization

System and methods for improved playback of a video stream are presented. Video snippets are identified that include a number of consecutive frames for playback. Snippets may be evenly temporally spaced in the video stream or may be content adaptive. Then the first frame of a snippet may be selected as the first frame of a scene or other appropriate stopping point. Scene detection, object detection, motion detection, video metadata, or other information generated during encoding or decoding of the video stream may aid in appropriate snippet selection.

Trick play in digital video streaming

Embodiments of the present invention provide techniques for coding video data efficiently based on detection of objects within video sequences. A video coder may perform object detection on the frame and when an object is detected, develop statistics of an area of the frame in which the object is located. The video coder may compare pixels adjacent to the object location to the object's statistics and may define an object region to include pixel blocks corresponding to the object's location and pixel blocks corresponding to adjacent pixels having similar statistics as the detected object. The coder may code the video frame according to a block-based compression algorithm wherein pixel blocks of the object region are coded according to coding parameters generating relatively high quality coding and pixel blocks outside the object region are coded according to coding parameters generating relatively lower quality coding.

Object detection informed encoding

For a first device participating in a video communications session with a second device, some embodiments provide a method that receives sets of encoding features supported by each of one or more decoders of the second device. The method selects an encoder for the video communication session from several encoders available to the first device. The method selects a set of encoding features of the selected encoder based on the sets of encoding features received from the second device. The method transmits an encoded video bitstream to the second device along with information identifying the selected features in order for the second device to select a decoder to decode the bitstream.

Selection of encoder and decoder for a video communications session

Codec techniques for fast switching with intermediate sequence

Video coders may perform perspective transformation of reference frames during coding in a manner that conserves processing resources. When a new input frame is available for coding, a camera position for the input frame may be estimated. A video coder may search for reference pictures having similar camera positions as the position of the input frame and, for each reference picture identified, the video coder may perform a prediction search to identify a reference picture that is the best prediction match for the input frame. Once the video coder identifies a reference picture to serve as a prediction source for the input frame, the video coder may derive a transform to match the reference frame data to the input frame data and may transform the reference picture accordingly. The video coder may code the input frame using the transformed reference picture as a prediction reference and may transmit coded frame data and the camera position of the input frame to a decoder. Thus, the video coder may perform derivation and execution of transforms on a limited basis which conserves system resources.

Predictive video coder with low power reference picture transformation

A system an method for determining to select frames from a video sequence that have high visual appeal and can be coded at high quality when frame rates of coded video drop to such low levels that perceptual sensations of moving video are lost. A metric is derived from a candidate input frame, and such metric is used to determine whether to increase or decrease a weight accorded to the candidate input frame. In an embodiment, the metric may be the auto-exposure data associated with the candidate input frame.

Frame encoding selection based on frame similarities and visual quality and interests

In a video coder/decoder system using variable resolution adaptation, decoder techniques provide a mechanism to changing resolution of coded lower-resolution video to a higher resolution for rendering. Coded video data of a low resolution frame may be decoded. A motion estimation search may be performed between the decoded low resolution frame and a cache of previously-stored high resolution video frames. If the motion estimation search generates one or more matches, high resolution video data of the decoded frame may generated as a derivation of matching data from the cached video frames.

Display quality in a variable resolution video coder/decoder system

An adaptive scaler switching system may implement multiple scalers including both a software scaler and a hardware scaler, and a controller that may manage the switch between scalers by considering the real-time constraints of the system and the available system resources. Information about the availability of system resources may be received in real-time, for example the controller may receive information about the system thermal status, the timing requirements for processing the video data, the quality of the scaled data, and any other relevant system statistics that may affect the scaler switch decision. According to an embodiment, the system may maintain statistics in a table, and update the table information as necessary.

Adaptive scaler switching

Video coding systems and methods protect against banding artifacts in decoded image content. According to the method, a video coder may identify, from content of pixel blocks of a frame of video data, which pixel blocks are likely to exhibit banding artifacts from the video coding/decoding processes. The video coder may assemble regions of the frame that are likely to exhibit banding artifacts based on the identified pixel blocks' locations with respect to each other. The video coder may apply anti-banding processing to pixel blocks within one or more of the identified regions and, thereafter, may code the processed frame by a compression operation.

Signal shaping techniques for video data that is susceptible to banding artifacts

Techniques for adjusting exposure parameters of a camera such that video data captured by the camera may be coded efficiently. A camera with auto exposure control may capture and output frames of video. A pre-processor may estimate brightness of the frames of the video output from the camera. A controller may estimate a rate of brightness change among the frames, and when the rate of change is lower than a predetermined threshold, the controller may reduce sensitivity of the auto exposure control. A coding engine may predictively code the video.

Adaptive auto exposure adjustment

In certain low-light or other poor image capture conditions, the camera capture settings may create noise fitting a known profile, including certain specific color ranges and signal frequencies, in the captured image. A pre-processor may identify frequency bands and DC offsets that may indicate noise in an image captured with known settings. Then the areas of the image containing the identified frequencies and offsets may be analyzed spatially and temporally to confirm that the identified regions are noise and not part of the scene and an appropriate filter defined.

Signal shaping for improved mobile video communication

Embodiments of the invention provide techniques for upsampling a video sequence for coding. According to the method, an estimate of camera motion may be obtained from motion sensor data. Video data may be analyzed to detect motion within frames output from a camera that is not induced by the camera motion. When non-camera motion falls within a predetermined operational limit, video upsampling processes may be engaged. In another embodiment, video upsampling may be performed by twice estimating image content for a hypothetical new a frame using two different sources as inputs. A determination may be made whether the two estimates of the frame match each other sufficiently well. If so, the two estimates may be merged to yield a final estimated frame and the new frame may be integrated into a stream of video data.

Growing community of inventors

San Jose, CA, United States of America

Douglas Scott Price