Sony Group Corporation

Seunghan Kim

Jong Hwa Lee

Gary Lyons

Praggya Garg

Luis Vasconcelos

Moises Tafolla

Allison Langley

Alexander Stoneham

An electronic device receives a set of 2D images comprising at least first 2D image of an object of interest and detects a plurality of 2D landmarks on the first 2D image. The detected plurality of 2D landmarks corresponds to shape-features of the object of interest. The electronic device aligns a first three-dimensional (3D) shape model of a reference object by fitting 2D landmarks of the first 3D shape model to the detected plurality of 2D landmarks. The electronic device estimates texture mapping information between the object of interest and the aligned first 3D shape model and generates a dataset by including the first 2D image, the aligned first 3D shape model, and the estimated texture mapping information as a first sample of the dataset. Based on the generated dataset, the electronic device trains a neural network model on a task of 3D reconstruction from a single 2D image.

3D dataset generation for neural network model training

An electronic device and method of dataset cleaning is provided. The electronic device receives a dataset comprising a plurality of samples, of which a first sample comprises a 2D image of an object of interest and a 3D shape model of the object of interest. The electronic device determines 2D landmarks from the 2D image and extracts 3D landmarks from the 3D shape model. The electronic device computes an error between the determined 2D landmarks and corresponding 2D locations of the extracted 3D landmarks on the 2D image, based on an error metric. Thereafter, the electronic device determines the computed error to be above a threshold. Based on the determination that the computed error is above the threshold, the electronic device updates the dataset by a removal of the first sample from the dataset and trains a neural network on a task of 3D reconstruction, based on the updated dataset.

Cleaning dataset for neural network training

A system and method for node-based image colorization is provided. The system receives a first user input via a Graphical User Interface (GUI) of an image/video editing application and controls a display device to display a node graph on the GUI based on the first user input. The node graph includes an input node to select grayscale images and a colorization node which may represent a workflow for colorization of at least a first object in the grayscale images. The system receives a second user input associated with a setting of the colorization node and selects an image colorization model based on the second user input. The system executes the workflow associated with the colorization node to transmit the grayscale images to a computing device that hosts the selected image colorization model, and to receive colorized images including at least the first object colorized based on a color effect.

Node-based image colorization on image/video editing applications

A computing system and method for neural network-based image colorization is provided. The computing system obtains a reference color image by selective application of a color effect on a region of interest of an input image and controls a display device to display a first node graph on a Graphical User Interface of an image/video editing application. The first node graph includes a colorization node representing a first workflow for colorization of at least a first object in grayscale images of a first image feed. The computing system selects the reference color image based on a user input and executes the first workflow associated with the colorization node by feeding the reference color image and the first image feed as an input to a neural network-based colorization model. The computing system receives a second image feed comprising colorized images as output of the neural network-based colorization model for the input.

Growing community of inventors

San Diego, CA, United States of America

Seunghan Kim