Broadcom Corporation

Chris Boross

Jeyhan Karaoguz

Xuemin Chen

Nambi Seshadri

Nambirajan Seshadri

A monoscopic video camera may capture, via at least one image sensor, two-dimensional video, and may capture, via at least one depth sensor, corresponding depth information for the captured two-dimensional video. The monoscopic video camera may then configure color related processing of video information corresponding to objects in the captured 2D video based on the captured corresponding depth information. Configuring color processing may comprise setting and/or adjusting color information associated with the objects. In this regard, setting and/or adjusting color information may be based on shadow, shade, and/or edge related information associated with the objects, which may be generated and/or modified based on the captured corresponding depth information based on the captured corresponding depth information. Configuring color processing may be based on surrounding regions adjacent to the objects. The surrounding regions adjacent to the objects may be determined based on the captured corresponding depth information.

Method and system for utilizing image sensor pipeline (ISP) for enhancing color of the 3D image utilizing z-depth information

A monoscopic camera comprising one or more image sensors and a depth sensor may generate video based on two-dimensional image data captured via the one or more image sensors and corresponding depth information captured via the depth sensor. The camera may process corresponding audio for the generated video based on the captured depth information. The audio processing may comprise mitigating noise in the corresponding audio, enhancing voice quality in the corresponding audio, and/or enhancing audio quality of the corresponding audio. The camera may be operable to determine, based on the captured depth information, one or more sound paths between a source of the corresponding audio and a microphone utilized to capture the corresponding audio emanating from the source. The processing of the audio may comprise removing portions of the captured audio arriving at the microphone via one or more reflection paths.

Method and system for noise cancellation and audio enhancement based on captured depth information

A monoscopic video camera may capture, via at least one image sensor, two-dimensional video, and may capture, via at least one depth sensor, corresponding depth information for the captured two-dimensional video. The monoscopic video camera may then adaptively configure scaling operations applicable to the captured two-dimensional video based on the depth information, which may comprise variably scaling different portions of the two-dimensional video. In this regard, the monoscopic video camera may determine, based on the depth information, a plurality of depth planes. The different portions of the two-dimensional video that are subjected to variable scaling may be determined based on the plurality of depth planes. Configuration of scaling operations may be performed in response to user input, which may comprise a zoom command. In this regard, scaling operations may be configured to focus on one or more of the different portions of the two-dimensional video based on zoom commands.

Method and system for utilizing image sensor pipeline (ISP) for scaling 3D images based on Z-depth information

Image sensors and a depth sensor of a monoscopic video sensing device are utilized to capture a 2D video and corresponding depth information. Regions of interest (ROIs) for the captured 2D video are selected based on the captured corresponding depth information. The monoscopic video sensing device selectively processes the captured 2D video and the captured corresponding depth information based on the selected ROIs. A 3D video is composed from the processed 2D video for display. The captured depth information that is synchronized to the captured 2D video is stored as metadata, and may be interpolated to match video resolution of the captured 2D video. The captured 2D video and the captured corresponding depth information are enhanced through scalable video coding. With 3D video rendering, relevant image/video components of the captured 2D video are selected based on the selected ROIs to compose the 3D video together with the corresponding depth information.

Method and system for creating a 3D video from a monoscopic 2D video and corresponding depth information

A mobile device, which comprises a monoscopic three-dimensional (3D) video generation device, one or more depth sensors and one or more devices operable to determine distance, may be operable to capture two-dimensional (2D) video image data and corresponding depth information of surroundings of the mobile device. The mobile device may capture distance information via the one or more devices operable to determine distance. A 3D map may be rendered by the mobile device utilizing the captured 2D video image data, the captured corresponding depth information and/or the captured distance information. The monoscopic 3D video generation device may comprise one or more image sensors. The one or more devices operable to determine distance may comprise an optical and/or electromagnetic emitting device. The 3D map may be generated by the mobile device. The 3D map may also be generated by the mobile device based on information received from a location server.

Method and system for utilizing depth information for generating 3D maps

A monoscopic 3D video generation device, which comprises one or more depth sensors, may be operable to store captured 2D video image data as a base layer and store captured corresponding depth information separately as an enhancement layer. The 2D video image data may be captured via one or more image sensors and the corresponding depth information may be captured via the one or more depth sensors in the monoscopic 3D video generation device. The monoscopic 3D video generation device may determine whether to transmit the enhancement layer to a video rendering device. The monoscopic 3D video generation device may encode the base layer. The monoscopic 3D video generation device may encode the enhancement layer based on the determination of transmitting the enhancement layer. The encoded base layer and/or the encoded enhancement layer may be transmitted to the video rendering device for 3D video rendering and/or 2D video rendering.

Method and system for utilizing depth information as an enhancement layer

A three-dimensional (3D) video rendering device monitors 3D effects associated with an object in a received 3D video image-by-image. The object may be re-located to a preferred location to adjust the associated 3D effects. Two-dimensional (2D) image data and corresponding depth information for the object at the current location are interpolated to the preferred location. A location difference and lighting condition changes corresponding to the re-location of the object are calculated to determine a view angle and lighting conditions for the object at the preferred location. 2D image data and depth information for the object at the preferred location are estimated based on the determined view angle and the determined lighting conditions for the object at the preferred location. The estimated 2D image data and the estimated corresponding depth information may be applied to the object at the preferred location to enhance the associated 3D effects for 3D video rendering.

Growing community of inventors

Sunnyvale, CA, United States of America

Chris Boross