Plantronics, Inc.

Hewlett-packard Development Company, L.P.

Polycom, Inc.

Dac International Inc.

Jian David Wang

Yong Yan

Varun Ajay Kulkarni

Stephen C Botzko

John Paul Spearman

Xiangdong Wang

Peter L Chu

David A Bryan

Rajen Bhatt

Kui Zhang

Meng Wang

Li Qu

Described are multiple cameras in a conference room, each pointed in a different direction. A primary camera includes a microphone array to perform sound source localization (SSL). The SSL is used in combination with a video image to identify the speaker from among multiple individuals that appear in the video image. Pose information of the speaker is developed. Pose information of each individual identified in each other camera is developed. The speaker pose information is compared to the pose information of the individuals from the other cameras. The best match for each other camera is selected as the speaker in that camera. The speaker views of each camera are compared to determine the speaker view with the most frontal view of the speaker. That camera is selected to provide the video for provision to the far end.

Matching active speaker pose between two cameras

A video conference call system is provided with a camera to generate an input frame image of a conference room, where the video conference call system detects a human head for each meeting participant captured in the input frame image by applying a machine learning human head detector model to said input image frame, generates a head bounding box which surrounds each detected human head and identifies a corresponding meeting participant, extracts a pixel width measure and pixel height measure from each head bounding box, and applies the extracted pixel width measure and pixel height measure to one or more reverse lookup tables to extract meeting room coordinates for each meeting participant identified by a corresponding head bounding box.

Estimation of human locations in two-dimensional coordinates using machine learning

Multiple cameras in a conference room, each pointed in a different direction and including a microphone array to perform sound source localization (SSL). The SSL is used in combination with the video image to identify the speaker from among multiple individuals that appear in the video image. Neural network or machine learning processing is performed on the identified speaker to determine the quality of the front or facial view of the speaker. The best view of the speaker's face from the various cameras is selected to be provided to the far end. If no view is satisfactory, a default view is selected and that is provided to the far end. The use of the SSL allows selection of the proper individual from a group of individuals in the conference room, so that only the speaker's head is analyzed for the best facial view and then framed for transmission.

Intelligent multi-camera switching with machine learning

The development of a region of interest (ROI) video frame that includes only ROIs of interest and not other elements and providing the ROI video frames in a single video stream simplifies the development of gallery view continuous presence displays. ROI position and size information metadata can be provided or subpicture concepts of the particular codec can be used to separate the ROIs in the ROI video frame. Metadata can provide perspective/distortion correction values, speaker status and any other information desired about the participant or other ROI, such as name. Only a single encoder and a single decoder is needed, simplifying both transmitting and receiving endpoints. Only a single video stream is needed, reducing bandwidth requirements. As each participant can be individually isolated, the participants can be provided in similar sizes and laid out as desired in a continuous presence display that is pleasing to view.

Representation and compression of gallery view for video conferencing

Correcting encode bit rate errors developed by a hardware encoder with an outer software rate adjustment loop. The rate adjustment loop maintains a VBV (video buffering verifier) buffer model. When VBV buffer fullness is over a threshold, the rate adjustment loop drops frames. If below the threshold, periodically a rate adjustment analysis performed. If a number of frames above a threshold have been dropped, the hardware encoder bit rate is decreased. A leaky bucket is checked to determine in bits and out bits. If the number of in bits exceeds the number of out bits times a multiplier, the hardware encoder bit rate is decreased. If the number of in bits times a multiplier is less than the number of out bits, the hardware encoder bit rate is increased. Dropping frames on buffer condition provides shorter term correction and adjustment of the hardware encoder bit rate provides longer term correction.

Adaptive rate control adjustment for hardware encoder

Utilizing two LTR frames for improved error recovery. By using two LTR frames, much better performance is achieved in terms of error recovery as the likelihood of the decoder having one of the two LTR frames is very high. When the decoder determines a frame is lost, the decoder provides a fast update request (FUR). The FUR includes a listing of the LTR frames present at the decoder. With this indication of the LTR frames present at the decoder, the encoder utilizes one of the LTR frames, preferably the most recent, to use as a reference to send the next frame as a P frame. The P frame is sent with an indication of the LTR frame used as reference. The use of two LTR frames and the feedback of LTR frames present at the decoder allows the minimization of the use of I frames for error recovery.

Long-term reference for error recovery in video conferencing system

A video encoding method eliminates the need for back channel feedback by using long-term reference frames for recovering data transmission errors. A videoconferencing endpoint captures image data and designates a first frame as a long term reference (LTR) frame. Subsequent intra frames us the LTR frame for reference. When a new LTR frame is designated, subsequent frames use the newly designated LTR frame for reference only after it is determined that the newly designated LTR frame is fully operational to serve in that role.

Long-term reference for error recovery without back channel

Methods for coding the current frame of a video data bit stream are provided. Methods include, for example, creating and maintaining a global codebook of historic values for the macroblocks of previous frames; searching entries in the global codebook to find the best estimation of the current frame and calculate a measure of the difference between values for the two macroblocks; comparing the effectiveness of the estimation macroblock provided by the global codebook entry to the estimation provided by a motion compensation scheme; choosing the best estimation method and coding the current frame; and updating the global codebook based on the new coded macroblock.

Growing community of inventors

Burnaby, Canada

Jian David Wang