Apple Inc.

Dazhong Zhang

Stephen Olivier Lemay

Xiaosong Zhou

Hsi-Jung Wu

Chris Yoochang Chung

Jae Hoon Kim

Hang Yuan

Jiefu Zhai

Peikang Song

Christopher M Garrido

Ming Chen

Karthick Santhanam

Yan Yang

Yunfei Zheng

Patrick Miauton

Alexandros Michael Tourapis

Douglas Scott Price

Joe S Abuan

Berkat S Tung

Ian J Baird

Hyeonkuk Jeong

David L Biderman

Keith Walter Rauenbuehler

Ming Jin

Yeping Su

James O Normile

Haitao Guo

Elizabeth Caroline Cranfill

David William Singer

Beibei Wang

Daniel Ben Pollack

Ting Chen

Roberto Garcia

Sudeng Hu

Eric L Chien

Xiaohua Yang

David R Conrad

Xing Wen

Bess C Chan

Albert E Keinath

Xiaoxiao Zheng

Sorin Constantin Cismas

Jun Xin

Mei Guo

Roberto Garcia, Jr

Luciano Miguel Verger

Thomas P Devanneaux

Ke Zhang

Hsien-Po Shiang

David Daming Kuo

Andrew Yanowitz

Yao-Chung Lin

Jose A Lozano Hinojosa

Gurtej Singh G Chandok

Christian L Duvivier

Vu H Chiem

Vikrant Kasarabada

Ionut Hristodorescu

Kevin Arthur Robertson

Johnny Trenh

Gurtej Singh Chandok

Jingteng Xue

Chieh Lu

Davide Concion

Joshua L Hare

Komei Harada

David A Eldred

Sasanka T Vemuri

Jochen Christian Schmidt

Xiaoxia Sun

Jinbo Qiu

David D Kuo

Jinbo Qui

Igor Kobzar

Peter Shiang

Sarah K Herrlinger

Paul Seung Ho Chang

Giribalan Gopalan

Participant systems in an audiovisual (AV) conferencing can, in one embodiment, use methods to ensure that all participants have a common platform to support AV conferencing, such as a common codec, and also attempt to provide higher quality AV conferencing (e.g., better than the common codec) based on available bandwidth and other criteria. The participant systems can use the common codec as a fallback platform when bandwidth or other criteria dictate a reduction in the quality of the AV conferencing. Other embodiments are also disclosed.

Optimizing media experience in conferencing with diverse participants

In communication applications, aggregate source image data at a transmitter exceeds the data that is needed to display a rendering of a viewport at a receiver. Improved streaming techniques that include estimating a location of a viewport at a future time. According to such techniques, the viewport may represent a portion of an image from a multi-directional video to be displayed at the future time, and tile(s) of the image may be identified in which the viewport is estimated to be located. In these techniques, the image data of tile(s) in which the viewport is estimated to be located may be requested at a first service tier, and the other tile in which the viewport is not estimated to be located may be requested at a second service tier, lower than the first service tier.

Adaptive coding and streaming of multi-directional video

A device implementing the subject technology may include at least one processor configured to transmit an allocation request requesting allocation of a group communication session with a plurality of devices and receive an allocation response in response to the allocation request, the allocation response including credential information for the device to use to join the group communication session. The at least one processor may be further configured to transmit an allocation bind request with the credential information to join the group communication session using the credential information and receive an allocation bind success response in response to the allocation bind request, the allocation bind success response indicating that the device has joined the group communication session. The at least one processor may be further configured to provide a join notification to the plurality of devices via an intermediary device to notify that the device has joined the group communication session.

Multi-device communication management

Techniques are disclosed for coding video data predictively based on predictions made from spherical-domain projections of input pictures to be coded and reference pictures that are prediction candidates. Spherical projection of an input picture and the candidate reference pictures may be generated. Thereafter, a search may be conducted for a match between the spherical-domain representation of a pixel block to be coded and a spherical-domain representation of the reference picture. On a match, an offset may be determined between the spherical-domain representation of the pixel block to a matching portion of the of the reference picture in the spherical-domain representation. The spherical-domain offset may be transformed to a motion vector in a source-domain representation of the input picture, and the pixel block may be coded predictively with reference to a source-domain representation of the matching portion of the reference picture.

Sphere projected motion estimation/compensation and mode decision

Embodiments of the present disclosure provide systems and methods for perspective shifting in a video conferencing session. In one exemplary method, a video stream may be generated. A foreground element may be identified in a frame of the video stream and distinguished from a background element of the frame. Data may be received representing a viewing condition at a terminal that will display the generated video stream. The frame of the video stream may be modified based on the received data to shift of the foreground element relative to the background element. The modified video stream may be displayed at the displaying terminal.

Systems and methods for perspective shifting in video conferencing session

In an example method, a system receives a plurality of frames of a video, and generates a data structure representing the video and representing a plurality of temporal layers. Generating the data structure includes: (i) determining a plurality of quality levels for presenting the video, where each of the quality levels corresponds to a different respective sampling period for sampling the frames of the video, (ii) assigning, based on the sampling periods, each of the frames to a respective one of the temporal layers of the data structure, and (iii) indicating, in the data structure, one or more relationships between (a) at least one the frames assigned to at least one of the temporal layers of the data structure, and (b) at least another one of the frames assigned to at least another one of the temporal layers of the data structure. Further, the system outputs the data structure.

Encoding and decoding video content

Techniques are disclosed for coding video data in which frames from a video source are partitioned into a plurality of tiles of common size, and the tiles are coded as a virtual video sequence according to motion-compensated prediction, each tile treated as having respective temporal location of the virtual video sequence. The coding scheme permits relative allocation of coding resources to tiles that are likely to have greater significance in a video coding session, which may lead to certain tiles that have low complexity or low motion content to be skipped during coding of the tiles for select source frames. Moreover, coding of the tiles may be ordered to achieve low coding latencies during a coding session.

Efficient coding of source video sequences partitioned into tiles

Techniques are disclosed for coding and decoding video data using object recognition and object modeling as a basis of coding and error recovery. A video decoder may decode coded video data received from a channel. The video decoder may perform object recognition on decoded video data obtained therefrom, and, when an object is recognized in the decoded video data, the video decoder may generate a model representing the recognized object. It may store data representing the model locally. The video decoder may communicate the model data to an encoder, which may form a basis of error mitigation and recovery. The video decoder also may monitor deviation patterns in the object model and associated patterns in audio content; if/when video decoding is suspended due to operational errors, the video decoder may generate simulated video data by analyzing audio data received during the suspension period and developing video data from the data model and deviation(s) associated with patterns detected from the audio data.

Applications for decoder-side modeling of objects identified in decoded video data

Techniques are presented for modifying images of an object in video, for example to correct for lens distortion, or to beautify a face. These techniques include extracting and validating features of an object from a source video frame, tracking those features over time, estimating a pose of the object, modifying a 3D model of the object based on the features, and rendering a modified video frame based on the modified 3D model and modified intrinsic and extrinsic matrices. These techniques may be applied in real-time to an object in a sequence of video frames.

Real-time face and object manipulation

Methods and Systems disclosed to counteract spatial distortions introduced by imaging processes of multi-directional video frames, where objects may be projected to spherical or equirectangular representations. Techniques provided to invert the spatial distortions in video frames used as reference picture data in predictive coding, by spatially transforming the image content of the reference picture data before this image content is being used for the prediction of input video data in prediction-based coders and decoders.

Processing of equirectangular object data to compensate for distortion by spherical projections

Techniques are disclosed for overcoming communication lag between interactive operations among devices in a streaming session. According to the techniques, a first device streaming video content to a second device and an annotation is entered to a first frame being displayed at the second device, which is communicated back to the first device. Responsive to a communication that identifies the annotation, a first device may identify an element of video content from the first frame to which the annotation applies and determine whether the identified element is present in a second frame of video content currently displayed at the first terminal. If so, the first device may display the annotation with the second frame in a location where the identified element is present. If not, the first device may display the annotation via an alternate technique.

Techniques to overcome communication lag between terminals performing video mirroring and annotation operations

Techniques presented herein provide an improved relay user experience and improved management of scarce computing and network resources as the number of relay endpoints increases. A sourcing endpoint device may generate a media feed, such as video and/or audio feed, representing contribution from a conference participant. The sourcing endpoint device may generate a priority value for the media feed, and the priority value may be transmitted to other members of the relay along with the input feed. Priority values of the different relay participants may be used by other devices, for example, intermediate servers or receiving endpoint devices, to manage aspects of the relay. For example, a relay server may prune streams from select endpoint devices based on relative priority values received from those devices. Alternatively, receiving endpoint devices may alter presentation of received feeds based on their associated priority values.

Media feed prioritization for multi-party conferencing

Techniques are presented for managing for visual prominence of participants in a video conference, including conferences where participants communicate visually, such as with sign language. According to these techniques, a visual prominence indication of a participant in a video conference may be estimated, a video stream of the participant may be encoded, and the encoded video stream may be transmitted along with an indication of the estimated visual prominence to a receiving device in the video conference.

Gesture and prominence in video conferencing

Growing community of inventors

Milpitas, CA, United States of America

Dazhong Zhang