Sony Group Corporation

Sony Corporation

Jong Hwa Lee

Gary Lyons

Praggya Garg

Seunghan Kim

Kenji Suzuki

Brent Faust

Edward Theodore Winter

Luis Vasconcelos

Antti Myllykoski

Moises Tafolla

Allison Langley

Arun Chandrasekar

Joseph Moon

Alexander Stoneham

Xihe Zhang

Gareth White

Kevin Barnes

Matthew Wnuk

Francisco Costela

Shiwei Jin

An electronic apparatus and method for visual speech recognition based on connectionist temporal classification (CTC) loss is disclosed. The electronic apparatus receives a video that includes human speakers and generates a prediction corresponding to lip movements of the human speakers. The prediction is generated based on application of a Deep Neural Network (DNN) on the video and the DNN is trained using a CTC loss function. The electronic apparatus detects, based on the prediction, word boundaries in a sequence of characters that correspond to the lip movements and divides the video into a sequence of video clips based on the detection. Each video clip corresponds to a word spoken by the human speakers. The electronic apparatus generates a sequence of word predictions by processing the sequence of video clips and generates a sentence, or a phrase based on the generated sequence of word predictions.

Visual speech recognition based on connectionist temporal classification loss

A wire structure may have a loop of fishing line on each of four sides, with weights to lower the line, giving a loop on each side. These loops may be tied to a central fishing line, which a user may pull to close the loops. This may catch a crab if one is in the trap, allowing a user to bring the crab to a wharf or boat, for example. Releasing the fishing line may allow the weights to lower, which may open the loops and release the crab without the user needing to touch the trap near the crab, which may reduce the chance of being pinched by a crab claw.

Crab fishing trap

An electronic device and method for reverse image search is provided. The electronic device receives an image. The electronic device extracts, by a DNN model, a first set of image features associated with the image and generates a first feature vector based on the first set of image features. The electronic device extracts, by an image-feature detection model, a second set of image features associated with the image and generates a second feature vector based on the second set of image features. The electronic device generates a third feature vector based on combination of the first and second feature vectors. The electronic device determines a similarity metric between the third feature vector and a fourth feature vector of each of a set of pre-stored images and identifies a pre-stored image based on the similarity metric. The electronic device controls a display device to display information associated with the pre-stored image.

Reverse image search based on deep neural network (DNN) model and image-feature detection model

Implementations generally relate to evaluation of similar content-based images. In some implementations, a method includes receiving a first image, where the first image includes at least one first object. The method further includes receiving a second image, where the second image includes at least one second object. The method further includes computing a structural similarity index measure (SSIM) value based on the at least one first object and the at least one second object. The method further includes computing a scale invariant feature transform (SIFT) value based on the at least one first object and the at least one second object. The method further includes computing a histogram value based on the at least one first object and the at least one second object. The method further includes computing a similarity score based on the SSIM value, the SIFT value, and the histogram value.

Evaluation of similar content-based images

The scaled Peak Signal-to-Noise Ratio (PSNR) method scales PSNR to 1.00 or 100% from infinite decibel values. The PSNR describes the ratio between the maximum possible value (power) of a signal and the power of signal's noise. Noise affects the quality of its representation. Signals dynamically fluctuate from the smallest to largest possible quantity. The unsettled range to maximum from minimum values of the noise is unknown. Therefore, the PSNR usually forms on a logarithmic way (log) of describing a ratio, the decibel (dB). The dB is able to gauge the level of power, communication signal (intensity or voltage), sound pressure, and intensity of several other items including images.

Scaled PSNR for image quality assessment

An electronic device receives a set of 2D images comprising at least first 2D image of an object of interest and detects a plurality of 2D landmarks on the first 2D image. The detected plurality of 2D landmarks corresponds to shape-features of the object of interest. The electronic device aligns a first three-dimensional (3D) shape model of a reference object by fitting 2D landmarks of the first 3D shape model to the detected plurality of 2D landmarks. The electronic device estimates texture mapping information between the object of interest and the aligned first 3D shape model and generates a dataset by including the first 2D image, the aligned first 3D shape model, and the estimated texture mapping information as a first sample of the dataset. Based on the generated dataset, the electronic device trains a neural network model on a task of 3D reconstruction from a single 2D image.

3D dataset generation for neural network model training

An electronic device and method of dataset cleaning is provided. The electronic device receives a dataset comprising a plurality of samples, of which a first sample comprises a 2D image of an object of interest and a 3D shape model of the object of interest. The electronic device determines 2D landmarks from the 2D image and extracts 3D landmarks from the 3D shape model. The electronic device computes an error between the determined 2D landmarks and corresponding 2D locations of the extracted 3D landmarks on the 2D image, based on an error metric. Thereafter, the electronic device determines the computed error to be above a threshold. Based on the determination that the computed error is above the threshold, the electronic device updates the dataset by a removal of the first sample from the dataset and trains a neural network on a task of 3D reconstruction, based on the updated dataset.

Cleaning dataset for neural network training

3D motion sensor data from a sensor on an athletic implement such as a golf club is clipped around an impact event and the clip then reduced to its 2D components, which are combined into a single 2D image and provided to a machine learning algorithm to output an indication of a flaw in the motion (e.g., a flaw in a golf swing).

Transforming sports implement motion sensor data to two-dimensional image for analysis

A system is provided that stores a neural network model trained on a training dataset which indicates an association between first graphic information associated with one or more first objects and corresponding first plurality of depth images. The system receives second graphic information that corresponds to the one or more first objects. The system further applies the trained neural network model received on the second graphic information. The system predicts a first depth image from the first plurality of depth images based on the application of the trained neural network model on the received second graphic information. The system extracts first depth information from the predicted first depth image. The first depth information corresponds to the one or more first objects indicated by the second graphic information.

Neural network model based depth estimation

A system for generation of training dataset is provided. The system controls a depth sensor to capture, from a first viewpoint, a first image a first depth value associated with the first object. The system receives tracking information from a handheld device associated with the depth sensor, based on a movement of the handheld device and the depth sensor in a 3D space. The system generates graphic information corresponding to the first object based on the received tracking information. The graphic information includes the first object from a second viewpoint. The system calculates a second depth value associated with the first object, based on the graphic information. The system generates, for a neural network model, a training dataset which includes a first combination of the first image and the first depth value, and a second combination of second images corresponding to the graphic information and the second depth value.

Training dataset generation for depth measurement

Implementations generally relate to determining a kinematic sequence. In some implementations, a method includes obtaining a video of a person performing an action. The method further includes determining from the video a plurality of points associated with the person, where the determining of the plurality of points is performed for each frame of the video. The method further includes converting two-dimensional (2D) coordinates of the plurality of points to three-dimensional (3D) coordinates, where the converting is performed for each frame of the video. The method further includes determining a movement of the plurality of points based at least in part on the 3D coordinates.

Determining a kinematic sequence

A system and method for node-based image colorization is provided. The system receives a first user input via a Graphical User Interface (GUI) of an image/video editing application and controls a display device to display a node graph on the GUI based on the first user input. The node graph includes an input node to select grayscale images and a colorization node which may represent a workflow for colorization of at least a first object in the grayscale images. The system receives a second user input associated with a setting of the colorization node and selects an image colorization model based on the second user input. The system executes the workflow associated with the colorization node to transmit the grayscale images to a computing device that hosts the selected image colorization model, and to receive colorized images including at least the first object colorized based on a color effect.

Node-based image colorization on image/video editing applications

A computing system and method for neural network-based image colorization is provided. The computing system obtains a reference color image by selective application of a color effect on a region of interest of an input image and controls a display device to display a first node graph on a Graphical User Interface of an image/video editing application. The first node graph includes a colorization node representing a first workflow for colorization of at least a first object in grayscale images of a first image feed. The computing system selects the reference color image based on a user input and executes the first workflow associated with the colorization node by feeding the reference color image and the first image feed as an input to a neural network-based colorization model. The computing system receives a second image feed comprising colorized images as output of the neural network-based colorization model for the input.

Neural network-based image colorization on image/video editing applications

Implementations generally relate to determining golf swing characteristics. In some implementations, a method includes obtaining a video of a person performing an action. The method further includes determining from the video a plurality of points associated with the person, where the determining of the plurality of points is performed for each frame of the video. The method further includes determining a plurality of three-dimensional (3D) coordinates for each point of the plurality of points, where the determining of the 3D coordinates is performed for each frame of the video. The method further includes determining a movement of the plurality of points based at least in part on the 3D coordinates. The method further includes determining one or more movement errors based at least in part on the movement of the plurality of points.

Determining golf swing characteristics

Implementations generally provide a user interface for video annotation tools. In some implementations, a method includes obtaining at least one video of at least one object performing at least one action displaying one or more portions of the at least one video in a user interface. The method further includes displaying a plurality of annotation tracks in the user interface, where each annotation track of the plurality of annotation tracks is associated with one or more of the at least one object and the at least one action in the at least one video. The method further includes obtaining one or more annotations associated with the at least one video based on the plurality of annotation tracks.

Growing community of inventors

San Diego, CA, United States of America

Jong Hwa Lee