Freescale Semiconductor,inc.

Nxp Usa, Inc.

Plantronics, Inc.

Polycom, Inc.

Yong Yan

Yolanda Prieto

Zhongli He

Jian David Wang

Kui Zhang

Erez Steinberg

Yehuda Yitschak

Zhong Li He

Peter L Chu

Michael Horowitz

Richard Ronald Hall

David A Bryan

Stephen C Botzko

Arnold W Yanof

Xiangdong Wang

Varun Ajay Kulkarni

John Paul Spearman

David Young

Athar Shah

Gregory C Ng

Srirama Rao Garikipati

Rajeev Tiwari

Miles A Sakauye

Yaniv Klein

In a multi-camera videoconferencing configuration, the locations of each camera are known. By referencing a known object visible to each camera, a 3D coordinate system is developed, with the position and angle of each camera being associated with that 3D coordinate system. The locations of the conference participants in the 3D coordinate system are determined for each camera. Sound source localization (SSL) from one camera, generally a central camera, is used to determine the speaker. The pose of the speaker is then determined. From the pose and the known locations of the cameras, the camera with the best frontal view of the speaker is determined. The 3D coordinates of the speaker are then used to direct the determined camera to frame the speaker. If the face of the speaker is not sufficiently visible, the next best camera view is determined, and the speaker framed from that camera view.

System and method of speaker reidentification in a multiple camera setting conference room

Multiple cameras in a conference room, each pointed in a different direction and including a microphone array to perform sound source localization (SSL). The SSL is used in combination with the video image to identify the speaker from among multiple individuals that appear in the video image. Neural network or machine learning processing is performed on the identified speaker to determine the quality of the front or facial view of the speaker. The best view of the speaker's face from the various cameras is selected to be provided to the far end. If no view is satisfactory, a default view is selected and that is provided to the far end. The use of the SSL allows selection of the proper individual from a group of individuals in the conference room, so that only the speaker's head is analyzed for the best facial view and then framed for transmission.

Intelligent multi-camera switching with machine learning

The development of a region of interest (ROI) video frame that includes only ROIs of interest and not other elements and providing the ROI video frames in a single video stream simplifies the development of gallery view continuous presence displays. ROI position and size information metadata can be provided or subpicture concepts of the particular codec can be used to separate the ROIs in the ROI video frame. Metadata can provide perspective/distortion correction values, speaker status and any other information desired about the participant or other ROI, such as name. Only a single encoder and a single decoder is needed, simplifying both transmitting and receiving endpoints. Only a single video stream is needed, reducing bandwidth requirements. As each participant can be individually isolated, the participants can be provided in similar sizes and laid out as desired in a continuous presence display that is pleasing to view.

Representation and compression of gallery view for video conferencing

A method of scaling complexity of a video processing system including determining a power saving factor based on an operating parameter and adjusting processing of video information based on the power saving factor to reduce computation complexity. The operating parameter may include available power and/or available processing capacity. A method of complexity scalability for a video processing system using prioritized layered coding including determining a power saving factor based on one or more metrics, such as power capacity and/or available processing capacity, and reducing processing complexity of multiple prioritized coding functions in a predetermined order of priority based on the level of the power saving factor. A video processing system including a power management circuit which determines the power saving factor and a video encoder system which correspondingly adjusts computation complexity.

Scaling video processing complexity based on power savings factor

A video encoding method eliminates the need for back channel feedback by using long-term reference frames for recovering data transmission errors. A videoconferencing endpoint captures image data and designates a first frame as a long term reference (LTR) frame. Subsequent intra frames us the LTR frame for reference. When a new LTR frame is designated, subsequent frames use the newly designated LTR frame for reference only after it is determined that the newly designated LTR frame is fully operational to serve in that role.

Long-term reference for error recovery without back channel

A video processing system dynamically adjusts video processing prediction error reduction computations in accordance with the amount of motion represented in a set of image data and/or available memory resources to store compressed video data. In at least one embodiment, video processing system adjusts utilization of prediction error computational resources based on the size of a prediction error between a first set of image data, such as current set of image data being processed, and a reference set of image data relative to an amount of motion in a current set of image data. Additionally, in at least one embodiment, the video processing adjusts utilization of prediction error computation resources based upon a fullness level of a data buffer relative to the amount of motion in the current set of image data.

Resource efficient video processing via prediction error computational adjustments

A method of encoding a video frame is disclosed in which video slices of the video frame are initially encoded in parallel using both interframe encoding and intraframe encoding. Then, after a first predetermined minimum amount of the video frame has been encoded, the method includes periodically determining whether the amount of intraframe encoded information for the frame achieves a first threshold, and when the first threshold is achieved, encoding the remainder of the video frame using only intraframe encoding. The method may include determining whether a lower second threshold is achieved based on relative complexity of the frame and quantization. The method may include performing similar comparisons on a slice by slice basis in which any one or more of the processing devices skips motion estimation and interframe encoding for corresponding video slices. A video encoder is disclosed which includes multiple processing devices and a shared memory.

Video scene change detection and encoding complexity reduction in a video encoder system having multiple processing devices

Embodiments are disclosed for a system and method for parallel processing of video signals. A multi-core processor is used to establish a master-slave relationship between a first processing core and a plurality of individual processing cores. Shared memory is used to store data and control messages. A plurality of individual private memories are associated with each of the individual processing cores; and control logic is used to establish a master-slave protocol for using the plurality of individual cores to process video data. The master processing core is operable to balance the video data processing load among the individual slave processing cores.

System and method for parallel video processing in multicore devices

A video encoder and a decoder analyze the spatial content video data in an H.264 stream using the discrete cosine transform (DCT). Although the DCT is computed as part of the H.264 encoding process, it is not computed as part of the decoding process. Thus, one would compute the DCT of the video data after it has been reconstructed by the video decoder for video post-processing or enhanced video encoding. A method for accelerating the computation of the DCT at the decoder side when transmitting intra-mode macroblocks uses information computed by the encoder and transmitted as part of the H.264 video stream.

Video decoder plus a discrete cosine transform unit

A method for transmitting encoded video data includes determining an available space for transmitting an encoded video data in a last remaining information block of a video packet. The method further includes if the available space is sufficient, based on a comparison of the an estimated size of the encoded video data with the available space, then transmitting the encoded video data using the last remaining information block of the video packet. The method further includes if the available space is insufficient, changing a coding scheme associated with the encoded video data to transform the encoded video data into reduced size video data and transmitting the reduced size video data using the last remaining information block of the video packet.

Transmission of packet data

An image processing system is disclosed which uses gain information from an input image to determine a threshold value used to filter the input image. The gain information is indicative of the amount of illumination of the input image and thus the noise level. The image processing system includes an image processor, a converter and a filter. The image processor receives and processes first image information into second image information and extracts the gain information from the first image information. The converter converts the gain information into a filter threshold, which is used by the filter to filter the second image information to provide filtered image information. The converter may include a lookup table storing noise characteristic estimates or the threshold values. The threshold values may further be based on subband size. The filter may be a wavelet-based transform denoising filter.

Gain controlled threshold in denoising filter for image signal processing

A dyadic spatial down sampling filter having tap values configured according to a Kaiser window function a beta factor of approximately 2.5, having approximately 1.5 side lobes, and having a down sampling ratio of approximately 1.9. The dyadic spatial down sampling filter may have tap values [&#x2212;1, 17, 32, 17, &#x2212;1]/64. A dyadic spatial up sampling filter having tap values configured according to a Kaiser window function having a beta factor of approximately 1.5, having approximately 2 side lobes, and having an up sampling ratio of approximately 2. The dyadic spatial up sampling filter may have tap values [&#x2212;5.44, 0, 20.71, 33.46, 20.71, 0, &#x2212;5.44]/64.0, or tap values [&#x2212;5, 0, 21, 32, 21, 0, &#x2212;5]/64, or tap values [&#x2212;5, 21, 21, &#x2212;5]/32.

Dyadic spatial re-sampling filters for inter-layer texture predictions in scalable image processing

A method of making a coding mode decision for a current macroblock of a current video frame including determining an INTERSAD value, selecting at least one video characteristic associated with the current video frame, comparing the INTERSAD value with a corresponding range of each selected video characteristic, selecting interframe coding if the INTERSAD value is within the corresponding range of each selected video characteristic, performing intraframe prediction to provide an intraframe prediction macroblock and determining an INTRASAD value if the INTERSAD value is an outlier of any selected video characteristic, selecting intraframe coding if the INTERSAD value is greater than the INTRASAD value and otherwise selecting interframe coding. The video characteristics may be based on any combination of an average of interframe differential sums, a sum of absolute differences between each pixel value of the current macroblock and a mean pixel value of the current macroblock, and a quantization parameter.

Growing community of inventors

Austin, TX, United States of America

Yong Yan