Sony Corporation

Sony Electronics Inc

Sony Group Corporation

Sony Corporation of America

Nikolaos Georgis

Fredrik Carpio

Paul Jin Hwang

Adrian Crisan

Milton Massey Frazier

Frank Li-De Lin

Hiroaki Nishimura

Djung N Nguyen

James Kuch

Kiyoharu Sassa

Yi Zhou

Christopher Russo

Dennis Adams

Robert Cowie

Jonathan Hood

Grant Sabesky

Darion Felix Rodgers

Glenda McFerran

Carolyn A Lynch

Justin Kauffman

Keisuke Omori

Gary Lyons

Edward Theodore Winter

Michiro Hirai

Luis Vasconcelos

Marc Birnkrant

Zhenglu Wu

Benjamin Lo

Chad Goolbis

Jason Vandekieft

Nobutaka Mitsuma

Luis Vanconcelos

An electronic apparatus and method for sequence stabilization of point cloud frames using motion information is disclosed. The electronic apparatus receives image data that includes images of objects. The image data corresponds to a duration in which in the objects are in a dynamic state. Based on the image data, the electronic apparatus generates a point cloud sequence and extracts motion information associated with the objects. The electronic apparatus determines a first set of 3D points of a first point cloud frame that is in a static state with respect to a second set of 3D points of a second point cloud frame, based on the motion information. The first and second point cloud frames are consecutive frames of point cloud sequence. The electronic apparatus further determines a difference between the first and the second set of 3D points and updates the first point cloud frame based on the difference.

Sequence stabilization of 3D point cloud frames using motion information

An apparatus includes one or more processors and logic encoded in one or more non-transitory media for execution by the one or more processors and when executed operable to receive a first image of an object, wherein the first image was captured by a first camera in a set of cameras. The logic is further operable to determine first keypoints and a first pose for the object based on the first image. The logic is further operable to receive a second image from a second camera in the set of cameras. The logic is further operable to determine an overlap of the first image and the second image based on locations of the first keypoints, the second keypoints, the first pose, and the second pose. The logic is further operable to determine a synchronization error between the first image and the second image based on the overlap.

User interface for rolling shutter camera arrays and accurate volumetric capture workflows

A method and system for synchronization of image data is provided. A plurality of image-capture devices is controlled to acquire a plurality of video clips of at least one object. Each video clip of the plurality of video clips is acquired by a corresponding image-capture device of the plurality of image-capture devices in a moving state. From the plurality of image-capture devices, a set of sensor data is acquired. Such data corresponds to the plurality of image-capture devices and is associated with a movement of the plurality of image-capture devices. Thereafter, relative offsets between the set of sensor data are determined by using cross-correlation and matching frames in each of the plurality of video clips is further determined, based on the relative offsets.

Accelerometer-assisted frame-accurate synchronization of mobile camera arrays

An apparatus includes one or more processors and logic encoded in one or more non-transitory media for execution by the one or more processors and when executed operable to receive images from a set of cameras that are positioned in a plurality of levels around one or more objects, wherein the set of cameras are positioned at a same orientation and each image is captured by a rolling shutter and associated with a position and orientation of a corresponding camera from the set of cameras. The logic is further operable to, for each level of cameras generate a three-dimensional pointcloud based on the position and orientation of corresponding cameras and matched extracted features. The logic is further operable to merge the three-dimensional pointcloud for each level of the cameras to form a three-dimensional volumetric model of one or more objects.

Self-organizing rolling shutter camera arrays for low-cost, accurate volumetric capture workflows

A minimal volumetric 3D transmission implementation enables efficient transmission of a 3D model to a client device. A volumetric 3D model is generated using a camera rig to capture frames of a subject. A viewer is able to select a view of the subject. A system determines an optimal subset of cameras of the camera rig to utilize to capture frames to generate the volumetric 3D model based on the viewer's selected view. The volumetric 3D model is transmitted to the user device. If the user changes the view, the process repeats, and a new subset of cameras are selected to generate the volumetric 3D model at a different angle.

Minimal volumetric 3D on demand for efficient 5G transmission

A hybrid implementation enables sharing the processing of 3D data locally and remotely based on processing and bandwidth factors. The hybrid implementation is flexible in determining what information to process locally, what information to transmit to a remote system, and what information to process remotely. Based on the available bandwidth, computing power/availability locally and computing power/availability remotely, the hybrid implementation is able to direct the processing of the data. By performing some of the processing locally and some of the processing remotely, more efficient processing is able to be implemented.

Hybrid edge-cloud compression of volumetric 3D data for efficient 5G transmission

A method and system for synchronization of image data is provided. From a plurality of image-capture devices, image data including a plurality of image sequences is received. Each image sequence corresponds to one of the plurality of image-capture devices. A set of sensor data is further received from the plurality of image-capture devices. Each sensor data comprises at least one of an Inertial Measurement Unit (IMU) data and audio data. Based on the received set of sensor data, a match is determined between a first set of image frames of a first image sequence and a second set of image frames of a second image sequence. An offset is computed between the first set of image frames and the second set of image frames, based on the match and the first image sequence is synchronized with the second image sequence based on the computed offset.

Synchronization of multi-device image data using multimodal sensor data

Implementations generally relate to estimating grouped observations. In some implementations, a method includes detecting a license plate of a vehicle. The method further includes capturing a plurality of images of the license plate. The method further includes computing candidate regions associated with the license plate based on the images of the license plate. The method further includes computing candidate characters of the license plate based on the candidate regions and the images of the license plate. The method further includes selecting target characters from the candidate characters based on one or more predetermined selection criteria.

Bidirectional statistical consolidation of multi-modal grouped observations

An information processing apparatus includes an entry and leaving management unit configured to analyze a captured image and detect vehicle information included in the captured image, and in a case where there is reservation information of parking corresponding to the detected vehicle information, open a gate of a parking lot corresponding to the reservation information.

Information processing apparatus and method, and program

Efficient 5G transmission of volumetric data using 3D character rigging techniques is able to be achieved by capturing volumetric data using RGB, depth or other special purpose cameras or sensors, reconstructing in 3D and then performing character rigging. The ability to render future or novel frames using rigging is able to be used in order to predict future and past models. By sending the rigging vectors and the compressed errors of difference between actual and predicted models, higher compression rates of volumetric sequences are able to be achieved compared to traditional methods. The decoder on the other side of the 5G channel reverses the process to synthesize original frames.

Efficient 5G transmission of volumetric data using 3D character rigging techniques

A system and method for recognition of license plate numbers from Bayer-domain image data is provided. The system acquires Bayer-domain image data of a scene comprising a vehicle and inputs such image data to a Deep Neural Network (DNN) model trained for license plate detection (LPD) and license plate recognition (LPR). The system extracts an LPD result as an output of the DNN model. Based on the LPD result, the system extracts, from the input Bayer-domain image data, a region-of-interest (RoI) comprising a license plate image associated with the vehicle. The system inputs the extracted RoI to the DNN model and extracts, as an output of the DNN model, an LPR result which indicates a confidence of the DNN model in recognition of a license plate number of the vehicle from the input RoI. The system determines the license plate number based on the extracted LPR result.

Recognition of license plate numbers from Bayer-domain image data

An electronic device for programmable rig control for three-dimensional (3D) reconstruction is provided. The electronic device controls a first image sensor to capture a first image of a first scene which includes a set of subjects. The electronic device further feed a first input to a neural network. The electronic device further receives a first output from the neural network based on the fed first input. The electronic device further selects one or more image sensors based on the received first output. The electronic device further controls a first set of structures associated with the selected one or more image sensors to re-arrange a rig around a three-dimensional (3D) physical space. The electronic device further controls a first set of image sensors, in the re-arranged rig, to capture one or more images for generation of one or more three-dimensional (3D) models of a first subject in the 3D physical space.

Programmable rig control for three-dimensional (3D) reconstruction

An electronic apparatus and method for license plate recognition-based vehicle control is provided. The electronic apparatus controls at least one of a plurality of image capture devices to capture one or more images of one or more second vehicles different from a first vehicle. The electronic apparatus detects a license plate of a second vehicle of the one or more second vehicles in the captured one or more images. The electronic apparatus further determines a depth map of the detected license plate of the second vehicle of the one or more second vehicles with respect to the first vehicle. The electronic apparatus further detects one or more events related to the second vehicle based on the determined depth map and controls one or more operations of the first vehicle based on the detected one or more events related to the second vehicle.

License plate recognition based vehicle control

An information processing device and method for sharing of compressed training data for neural network training is provided. The information processing device receives a first image which includes an object of interest. The information processing device extracts, from the received first image, a region of interest which includes the object of interest. Once extracted, the extracted region of interest is provided to an input layer of N numbers of layers of a first neural network, trained on an object detection task. The information processing device selects an intermediate layer of the first neural network and extracts a first intermediate result as an output generated by the selected intermediate layer of the first neural network based on the input RoI. Once extracted, the information processing device shares the extracted first intermediate result as compressed training data with a server to train a second neural network on the object detection task.

Sharing of compressed training data for neural network training

An electronic device includes circuitry that determines probability map information for a first image, based on application of a neural network model on the first image. The neural network model is trained to detect one or more objects based on a plurality of images associated with the one or more objects. The probability map information indicates a probability value for each pixel in the first image. A region corresponding to the one or more objects is detected in the first image based on the probability map information. A first set of sub-images is determined from the detected region, based on application of a stochastic optimization function on the probability map information. The one or more objects are detected from a second set of sub-images of the first set of sub-images, based on application of the neural network model on the second set of sub-images.

Growing community of inventors

San Diego, CA, United States of America

Nikolaos Georgis