Apple Inc.

Jim Normile

Hyeonkuk Jeong

Joe S Abuan

Berkat S Tung

Ryan R Salsbury

Thomas Pun

Xin Tong

Hsi-Jung Wu

Barin Geoffry Haskell

Adriana Dumitras

Xiaojin Shi

Meriko Borogove

Hsi Jung Wu

Berkat Shing Tung

Some embodiments provide an architecture for establishing multi-participant video conferences. This architecture has a central distributor that receives video images from two or more participants. From the received images, the central distributor generates composite images that the central distributor transmits back to the participants. Each composite image includes a set of sub images, where each sub image belongs to one participant. In some embodiments, the central distributor saves network bandwidth by removing each particular participant's image from the composite image that the central distributor sends to the particular participant. In some embodiments, images received from each participant are arranged in the composite in a non-interleaved manner. For instance, in some embodiments, the composite image includes at most one sub-image for each participant, and no two sub-images are interleaved.

Video processing in a multi-participant video conference

Some embodiments provide an architecture for establishing a multi-participant conference. This architecture has one participant's computer in the conference act as a central content distributor for the conference. The central distributor receives data (e.g., video and/or audio streams) from the computer of each other participant, and distributes the received data to the computers of all participants. In some embodiments, the central distributor receives A/V data from the computers of the other participants. From such received data, the central distributor of some embodiments generates composite data (e.g., composite image data and/or composite audio data) that the central distributor distributes back to the participants.

Multi-participant conference setup

Techniques for encoding data based at least in part upon an awareness of the decoding complexity of the encoded data and the ability of a target decoder to decode the encoded data are disclosed. In some embodiments, a set of data is encoded based at least in part upon a state of a target decoder to which the encoded set of data is to be provided. In some embodiments, a set of data is encoded based at least in part upon the states of multiple decoders to which the encoded set of data is to be provided.

Complexity-aware encoding

A method of encoding a sequence of video images is described. The method receives the sequence of video images. The method iteratively examines different encoding solutions for the sequence of video images to identify an encoding solution that optimizes image quality while meeting a target bit rate and satisfying a set of constraints regarding flow of encoded data through an input buffer of a hypothetical reference decoder for decoding the encoded video sequence. The iterative examining includes, for each encoding solution, determining whether the hypothetical reference decoder underflows while processing the encoding solution for any set of images within the video sequence.

Multi-pass video encoding solution for buffer underflow

A novel multi-pass encoding method that encodes several images (e.g., several frames of a video sequence) is described. The method iteratively performs an encoding operation that encodes these images. The encoding operation is based on a nominal quantization parameter, which the method uses to compute quantization parameter for the images. During several different iterations of the encoding operation, the method uses several different nominal quantization parameters. The method stops its iterations when it reaches a terminating criterion (e.g., it identifies an acceptable encoding of the images).

Multi-pass video encoding based on different quantization parameters

Some embodiments provide an architecture for establishing a multi-participant conference. This architecture has one participant's computer in the conference act as a central content distributor for the conference. The central distributor receives data (e.g., video and/or audio streams) from the computer of each other participant, and distributes the received data to the computers of all participants. In some embodiments, the central distributor receives A/V data from the computers of the other participants. From such received data, the central distributor of some embodiments generates composite data (e.g., composite image data and/or composite audio data) that the central distributor distributes back to the participants. The central distributor in some embodiments can implement a heterogeneous audio/video conference. In such a conference, different participants can participate in the conference differently. For instance, different participants might use different audio or video codecs. Moreover, in some embodiments, one participant might participate in only the audio aspect of the conference, while another participant might participate in both audio and video aspects of the conference.

Heterogeneous video conferencing

Some embodiments provide a method for adjusting a video sampling rate during an video encoding operation. The method receives a metric that quantifies the usage of a computer that performs the video encoding operation. The method computes an adjustment factor based on the metric. In some embodiments the metric is a system idle time. The method defines the video sampling rate based on the adjustment factor. Some embodiments compute the adjustment factor by performing a non-linear operation based on the metric. In some embodiments, the non-linear operation includes performing an integral operation based on the metric. In some embodiments, the non-linear operation includes performing a derivative operation based on the metric. Some embodiments perform more than one operation to compute the adjustment factor. In some embodiments, each operation is assigned a weight.

Growing community of inventors

Los Altos, CA, United States of America

Jim Normile