Personify, Inc.

University of Illinois

Nvidia Corporation

Pesonify, Inc.

Sanjay J Patel

Dennis Lin

Minh Do

Quang Nguyen

Sumant Kowshik

Daniel P Dabbelt

Monier Maher

Simion Venshtain

Christopher Frederick Lamb

Mert Dikmen

Long Truong

Jean Pierre Bordes

Peter Y Hsu

Long Dang

David Scherba

Greg Meyer

Elad Yarkony

Quang Il Nguyen

Benjamin Chidester

Quang H Nguyen

Daniel P Dabbeit

A system for background image subtraction includes a computing device coupled with a 3D video camera, a processor of the device programmed to receive a video feed from the camera containing images of one or more subject that include depth information. The processor, for an image: segments pixels and corresponding depth information to identify foreground regions associated with a target user.

System for background subtraction with 3D camera

Embodiments disclose systems and methods for transmitting user-extracted video and content more efficiently by recognizing that user-extracted video provides the potential to treat parts of a single frame of a user-extracted video differently. An alpha mask of the image part of the user-extracted video is used when encoding the image part so that it retains a higher quality upon transmission than the remainder of the user-extracted video.

Transmitting video and sharing content via a network

Disclosed herein are systems and methods for iterative adjustment of video-capture settings based on identified persona. In an embodiment, a method includes receiving video frames that are being captured by a video camera of an ongoing scene. The method also includes identifying a persona in one or more of the received frames at least in part by identifying, in each such frame, a set of pixels classified as being representative of the persona. The method also includes selecting, based collectively on the brightness values of the pixels in the identified set, a respective adjustment command for one or more adjustable video-capture settings of the camera, as well as outputting the selected commands to the camera for use in continuing to capture video data representative of the ongoing scene.

Systems and methods for iterative adjustment of video-capture settings based on identified persona

Disclosed herein are methods and systems for combining foreground video and background video using chromatic matching. In an embodiment, a system obtains foreground video data. The system obtains background video data. The system determines a color-distribution dimensionality of the background video data to be either high-dimensional chromatic or low-dimensional chromatic. The system selects a chromatic-adjustment technique from a set of chromatic-adjustment techniques based on the determined color-distribution dimensionality of the background video data. The system adjusts the foreground video data using the selected chromatic-adjustment technique. The system generates combined video data at least in part by combining the background video data with the adjusted foreground video data. The system outputs the combined video for display.

Methods and systems for combining foreground video and background video using chromatic matching

A system for background image subtraction includes a computing device coupled with a 3D video camera, a processor o&#xa3; the device programmed to receive a video feed from the camera containing images of one or more subject that include depth information. The processor, for an image: segments pixels and corresponding depth information into three different regions including foreground (FG), background (BG), and unclear (UC); categorizes UC pixels as FG or BG using a function that considers the color and background history (BGH) information associated with the UC pixels and the color and BGH information associated with pixels near the UC pixels; examines the pixels marked as FG and applies temporal and spatial filters to smooth boundaries of the FG regions; constructs a new image by overlaying the FG regions on top of a new background; displays a video feed of the new image in a display device; and continually maintains the BGH.

Embodiments disclose extracting a user persona from a video of arbitrary duration and associating that persona with text for a chat session. Embodiments cause the persona to be extracted at the moment text is sent or received to convey the body language associated with the text.

Integrating user personas with chat sessions

A system is disclosed for executing depth image-based rendering of a 3D image by a computer having a processor and that is coupled with one or more color cameras and at least one depth camera. The color cameras and the depth camera are positionable at different arbitrary locations relative to a scene to be rendered. In some examples, the depth camera is a low resolution camera and the color cameras are high resolution. The processor is programmed to propagate depth information from the depth camera to an image plane of each color camera to produce a propagated depth image at each respective color camera, to enhance the propagated depth image at each color camera with the color and propagated depth information thereof to produce corresponding enhanced depth images, and to render a complete, viewable image from one or more enhanced depth images from the color cameras. The processor may be a graphics processing unit.

System for executing 3D propagation for depth image-based rendering

A color image and a depth image of a live video are received. Each of the color image and the depth image are processed to identify the foreground and the background of the live video. The background of the live video is removed in order to create a foreground video that comprises the foreground of the live video. A control input may be received to control the embedding of the foreground video into a second background from a background feed. The background feed may also comprise virtual objects such that the foreground video may interact with the virtual objects.

Systems and methods for embedding a foreground video into a background feed based on a control input

A system for background image subtraction includes a computing device coupled with a 3D video camera, a processor of the device programmed to receive a video feed from the camera containing images of one or more subject that include depth information. The processor, for an image: segments pixels and corresponding depth information into three different regions including foreground (FG), background (BG), and unclear (UC); categorizes UC pixels as FG or BG using a function that considers the color and background history (BGH) information associated with the UC pixels and the color and BGH information associated with pixels near the UC pixels; examines the pixels marked as FG and applies temporal and spatial filters to smooth boundaries of the FG regions; constructs a new image by overlaying the FG regions on top of a new background; displays a video feed of the new image in a display device; and continually maintains the BGH.

Disclosed herein are systems and methods for iterative adjustment of video-capture settings based on identified persona. In an embodiment, a method includes receiving video frames being captured by a video camera of an ongoing scene. The method also includes identifying a persona in one or more of the received frames at least in part by identifying, in each such frame, a set of pixels that is representative of the persona in the frame and that does not include any pixels representative of a background of the frame. The method also includes selecting, based collectively on the brightness values of the pixels in the identified set of pixels of one or more frames, an adjustment command for one or more adjustable video-capture settings of the camera, as well as outputting the selected commands to the camera for use in continuing to capture video data representative of the ongoing scene.

Transmitting video and sharing content via a network using multiple encoding techniques

A method, apparatus, system, and computer program product for of digital imaging. Multiple cameras comprising lenses and digital images sensors are used to capture multiple images of the same subject, and process the multiple images using difference information (e.g., an image disparity map, an image depth map, etc.). The processing commences by receiving a plurality of image pixels from at least one first image sensor, wherein the first image sensor captures a first image of a first color, receives a stereo image of the first color, and also receives other images of other colors. Having the stereo imagery, then constructing a disparity map and an associated depth map by searching for pixel correspondences between the first image and the stereo image. Using the constructed disparity map, captured images are converted into converted images, which are then combined with the first image, resulting in a fused multi-channel color image.

System and method for generating a depth map and fusing images from a camera array

A color image and a depth image of a live video are received. A user is extracted from the information of the color image and the depth image. Spurious depth vales may be corrected. Points or pixels of an image as seen from a viewpoint of a reference camera at a reference camera location are mapped to points of the image as would be seen from a viewpoint of a virtual camera at a virtual camera location. As such, a transformed color image is generated. Disoccluded pixels may be processed to address any gaps within the transformed color image.

Systems and methods for generating a virtual camera viewpoint for an image

A system and method is disclosed for extracting a user persona from a video and embedding that persona into a background feed that may have other content, such as text, graphics, or additional video content. The extracted video and background feed are combined to create a composite video that comprises the display in a videoconference. Embodiments cause the user persona to be embedded at preset positions, or in preset formats, or both, depending on the configuration, position, or motion of the user's body.

Growing community of inventors

St. Louis, MO, United States of America

Sanjay J Patel