Sony Corporation

Sony Group Corporation

Sony Corporation of America

Hiroyuki Takeda

Mohammad Gharavi-Alkhansari

Alexander Berestov

Tamaki Kojima

Golnaz Abdollahian

Daniel A Usikov

Kendai Furukawa

Wei Ji

Kenji Tashiro

Mohammad Gharavi Alkhansari

Suren Deepak Rajasekaran

Daniel Usikov

A system and method for neural-network based image-to-image translation is provided. The system includes a generator network which receives an image and generates a first synthetic image based on the received image. The first synthetic image is different from the image in appearance, style, or domain. The system includes a feature-to-image generator that extracts a feature map from an intermediate layer of the generator network and generates a second synthetic image. The system further includes a first discriminator network that generates a result indicative of whether the first synthetic image is real or fake in comparison to a first reference image and a second discriminator network that generates a result indicative of whether the second synthetic image is real or fake in comparison to a second reference image. The system further includes a processor which trains the first generator network in adversarial manner based on losses associated with the results.

Neural network-based image-to-image translation

An integrated photo-video volumetric capture system for 3D/4D scanning acquires 3D scans and 4D scans by acquiring images and videos simultaneously. The volumetric capture system for high-quality 4D scanning and mesh-tracking is used to establish topology correspondences across a 4D scanned mesh sequence for generating corrective shapes which will be used in shape interpolation and skeleton driven deformation. The volumetric capture system aids mesh-tracking for maintaining mesh registration (topology consistency) along with ease of extreme pose modeling. Major upper body and lower body joints are able to be identified that are important for generating deformation and capturing the same using a wide range of motion for all movement types across all joint categories. By using the volumetric capture system and mesh tracking, the topology changes are tracked. Each pose captured will have the same topology which makes blending between multiple poses easier and more accurate.

Automatic blending of human facial expression and full-body poses for dynamic digital human model creation using integrated photo-video volumetric capture system and mesh-tracking

An electronic device and method for 3D face modeling based on neural networks is provided. The electronic device receives a two-dimensional (2D) color image of a human face with a first face expression and obtains a first three-dimensional (3D) mesh of the human face with the first face expression based on the received 2D color image. The electronic device generates first texture information and a first set of displacement maps. The electronic device feeds the generated first texture information and the first set of displacement maps as an input to the neural network and receives an output of the neural network for the fed input. Thereafter, the electronic device generates a second 3D mesh of the human face with a second face expression which is different from the first face expression based on the received output.

3D face modeling based on neural networks

An electronic device and method of preservation of geometry details in tracked meshes is provided. A set of three-dimensional (3D) scans and a sequence of tracked meshes of an object of interest are acquired. The sequence of tracked meshes comprises a set of tracked meshes which temporally corresponds to the set of 3D scans. Based on differences between the set of tracked meshes and the set of 3D scans, a set of displacement maps is generated. A plurality of vectors is computed, each of which includes surface tension values associated with mesh vertices of a corresponding tracked mesh. Based on the set of displacement maps and a corresponding set of vectors, a neural network is trained on a task of displacement map generation. The trained neural network model is applied on the plurality of vectors to generate displacement maps. Each tracked mesh is updated based on a corresponding displacement map.

Preserving geometry details in a sequence of tracked meshes

An electronic apparatus and method for image-based mask frame interpolation is provided. The electronic apparatus receives a sequence of image frames, including a group of image frames and an intermediate image frame. The electronic apparatus generates foreground (FG) masks based on application of a trained Neural Network on the group of image frames and selects a Region-of-Interest (RoI) from a first image frame of the group of image frames. The electronic apparatus extracts first feature vectors for pixels within the selected RoI and second feature vector for a first pixel of the intermediate image frame. The electronic apparatus estimates a first mask value for the first pixel based on application of a statistical function on the first feature vectors and the second feature vector and generates an intermediate FG mask based on the estimated first mask value and re-estimation of mask values for remaining pixels of the intermediate image frame.

Image-based mask frame interpolation

An electronic device and method for image mask generation using a deep neural network is provided. The electronic device stores an input image that includes an object of interest generates a pilot mask for the input image by application of a trained deep neural network (DNN) on the input image. The electronic device extracts, from the input image, a region of interest that includes the object of interest, based on the generated pilot mask and resizes the extracted region of interest such that a size of the resized region of interest equals a threshold scaling size. The electronic device further generates a final mask by reapplication of the trained DNN on the resized region of interest.

Image mask generation using a deep neural network

An image processing apparatus and method for object segmentation in color image frames, is provided. The image-processing apparatus generates, based on a first neural network model, a first foreground mask and a second foreground mask for an object-of-interest in a first color image frame and an upcoming color image frame, respectively, of the sequence of color image frames. The image-processing apparatus determines a third foreground mask based on interpolation of the first foreground mask and the second foreground mask and updates, by use of the second neural network model, the third foreground mask to a fourth foreground mask. The image processing apparatus segments the object-of-interest from at least the first color image frame, the upcoming color image frame, and the intermediate color image frame by use of the first foreground mask, the second foreground mask, and the fourth foreground mask, respectively.

Multiple neural networks-based object segmentation in a sequence of color image frames

An object segmentation system that includes a first type of sensor, a second type of sensor and a control circuitry. The first-type of sensor captures a sequence of color image frames of a scene. The second-type of sensor captures a depth image for each corresponding color image frame of the sequence of color image frames. The control circuitry generates a point cloud for an input color image frame. The control circuitry segments a foreground human object from a background of the input color image frame. The control circuitry detects a ground plane for the scene captured in the input color image frame. The control circuitry recovers a feet region in a defined region from a level of the detected ground plane. The control circuitry extracts the foreground human object with the recovered feet region from the background of the input color image frame, based on the detection of ground plane.

Missing feet recovery of a human object from an image sequence based on ground plane detection

An image-processing apparatus includes a first-type of sensor, a second-type of sensor, and a control circuitry. The control circuitry receives an input color image frame of a sequence of color image frames from the first-type of sensor and a corresponding input depth image from the second-type of sensor. The control circuitry generates a first foreground mask of an object-of-interest for the input color image frame. The control circuitry then detects a first set of pixels in the generated first foreground mask as misclassified image pixels. The control circuitry updates the background color image and the background depth image of the scene. The control circuitry precisely extracts the object-of-interest from the input color image frame, based on at least the updated background color image and the background depth image of the scene.

Object segmentation in a sequence of color image frames by background image and background depth correction

An image processing apparatus includes a first type of sensor, a second type of sensor, and a control circuitry. The control circuitry receives a first foreground mask for an object-of-interest in a previous color image frame and further estimate a second foreground mask for the object-of-interest in a current color image frame at a first image resolution. The control circuitry further upsamples the second foreground mask to a second image resolution and select at least one region-of-interest in the current color image frame. The control circuitry then upscales the upsampled second foreground mask by an expectation parameter generated based on a kernel density-based estimation for the at least one pixel in a selected region-of-interest. The upscaled second foreground mask exhibits a minimum foreground mask error that is caused by upsampling with respect to the estimated second foreground mask.

Object segmentation in a sequence of color image frames based on adaptive foreground mask upsampling

Various aspects of a video-processing system and method for object detection in a sequence of image frames are disclosed herein. The system includes an image-processing device configured to receive a first object template for an object in a first image frame that includes one or more objects. A plurality of object candidates that corresponds to the object for a second image frame are determined by use of the shape of the received first object template. One of the determined plurality of object candidates is selected as a second object template, based on one or more parameters. The received first object template is updated to the selected second object template to enable segmentation of the object in the second image frame and/or subsequent image frames.

Video processing system and method for object detection in a sequence of image frames

An image-processing device and method for foreground mask correction for object segmentation, includes receipt of a sequence of image frames. A first FG mask is estimated using depth information associated with the input image frame and by binarizing a difference between the input image frame and a background (BG) image of the input image frame. A first set of pixels with a first mask value to be updated to a second mask value is identified in a boundary region of the estimated first FG mask. A second FG mask is determined based on the identification of the first set of pixels in the boundary region of the estimated first FG mask.

Image-Processing device and method for foreground mask correction for object segmentation

Segmentation is the process of partitioning an image into regions under certain rules. One implementation is to separate human objects that appear in a sequence of images (video) from the background. The goal is to find humans and segment them out in real-time, fully automatic (with no user input), and the result is produced immediately after a new image is captured. Once the segmentation process is started, the method is constantly learning (or updating) the decision rule for segmenting out human objects from the background by itself. Moving object detection by a Kalman filter-based approach roughly detects the region where moving objects are present.

Method for moving object detection by a Kalman filter-based approach

Various aspects of an image processing system and method for object boundary smoothening for image segmentation, includes receipt of a user input to enable selection of a foreground object in an input color image. A frequency of occurrence of foreground pixels with respect to background pixels is determined for a plurality of pixels within a local pixel analysis window. The local pixel analysis window is positioned in a first region of the input color image to encompass at least a first pixel to be validated for a correct mask value. A first cost value and a second cost value is selected for the first pixel based on the determined frequency of occurrence of the foreground pixels. An object boundary is generated for a portion of the foreground object based on the selected first cost value and the second cost value for the first pixel.

Image processing system and method for object boundary smoothening for image segmentation

Automatic scaling of image objects being moved or copied to a target object location is presented. An original image and its depth map are received within which the user selects an original object and a target location for the object to be moved or copied. A center of scaling is found and a base location selected at which depth value is to be obtained at the target location for use in scaling. Target object is then scaled in response to a ratio between original object depth and target object depth. An occlusion check is made, after which non-occluded pixels are pasted to the target location to complete the move/copy which is in proper proportion and accounts for surrounding structures.

Growing community of inventors

San Jose, CA, United States of America

Hiroyuki Takeda