Imperial College Innovations Limited

Imperial College of Science, Technology and Medicine

Andrew Davison

Michael Bloesch

Stefan Leutenegger

Stephen Lloyd James

Edgar Antonio Sucar Escamilla

Daniel James Lenton

Kentaro Wada

Robert Lukierski

John Brendan Mccormac

Tristan William Laidlow

Ankur Handa

Ronald Clark

Thomas Whelan

Renato Fernando Salas Moreno

Ben Glocker

Shuaifeng Zhi

A computer-implemented method of determining a pose of each of a plurality of objects includes, for each given object: using image data and associated depth information to estimate a pose of the given object. The method includes iteratively updating the estimated poses by: sampling, for each given object, a plurality of points from a predetermined model of the given object transformed in accordance with the estimated pose of the given object; determining first occupancy data for each given object dependent on positions of the points sampled from the predetermined model, relative to a voxel grid containing the given object; determining second occupancy data for each given object dependent on positions of the points sampled from the predetermined models of the other objects, relative to the voxel grid containing the given object; and updating the estimated poses of the plurality of objects to reduce an occupancy penalty.

Image processing system and method

A computer-implemented method of estimating a pose of a target object in a three-dimensional scene includes: obtaining image data and associated depth information representing a view of the three-dimensional scene; processing the image data and the associated depth information to generate a volumetric reconstruction for each of a plurality of objects in the three-dimensional scene, including the target object; determining a volumetric grid containing the target object; generating, using the generated volumetric reconstructions, occupancy data indicating portions of the volumetric grid occupied by free space and portions of the volumetric grid occupied by objects other than the target object; and estimating the pose of the target object using the generated occupancy data and pointwise feature data for a plurality of points on a surface of the target object.

Certain examples described herein relate to a system for processing image data. In such examples, the system includes an input interface to receive the image data, which is representative of at least one view of a scene. The system also includes an initialisation engine to generate a first latent representation associated with a first segmentation of at least a first view of the scene, wherein the first segmentation is a semantic segmentation. The initialisation engine is also arranged to generate a second latent representation associated with at least a second view of the scene. The system additionally includes an optimisation engine to jointly optimise the first latent representation and the second latent representation, in a latent space, to obtain an optimised first latent representation and an optimised second latent representation.

Scene representation using image processing

An image processing system configured to obtain a mesh representation of a scene, wherein the mesh representation comprises a plurality of polygons defined by respective vertices associated with an in-plane position, the in-plane position being in a plane comprising a first dimension and a second dimension, and the vertices having an associated vertex depth value in a third dimension different from the first dimension and the second dimension. The image processing system comprises an in-plane position estimation network configured to process image data representative of an image of the scene to estimate the in-plane positions associated with respective vertices of the mesh representation. The image processing system further comprises a depth estimation engine configured to process the in-plane positions and the image data to estimate the associated vertex depth values for the respective vertices of the mesh representation.

A method comprising applying an object recognition pipeline to frames of video data. The object recognition pipeline provides a mask output of objects detected in the frames. The method includes fusing the mask output of the object recognition pipeline with depth data associated with the frames of video data to generate a map of object instances, including projecting the mask output to a model space for the map of object instances using a camera pose estimate and the depth data. An object instance in the map of object instances is defined using surface-distance metric values within a three-dimensional object volume, and has an object pose estimate indicating a transformation of the object instance to the model space. The object pose estimate and the camera pose estimate form nodes of a pose graph for the map of model instances.

Mapping object instances using video data

A control system for a robotic device comprising a task embedding network to receive one or more demonstrations of a task and to generate a task embedding. The task embedding comprises a representation of the task, and each demonstration comprises one or more observations of a performance of the task. The control system includes a control network to receive the task embedding from the task embedding network and to apply a policy to map a plurality of successive observations of the robotic device to respective control instructions for the robotic device. The policy applied by the control network is modulated across the plurality of successive observations of the robotic device using the task embedding from the task embedding network.

Device control using policy training based on task embeddings

Certain examples described herein relate to estimating dimensions of an enclosed space such as a room using a monocular multi-directional camera device. In examples, a movement of the camera device around a point in a plane of movement is performed, such as by a robotic device. Using the monocular multi-directional camera device, a sequence of images are obtained at a plurality of different angular positions during the movement. Pose data is determined from the sequence of images. The pose data is determined using a set of features detected within the sequence of images. Depth values are then estimated by evaluating a volumetric function of the sequence of images and the pose data. A three dimensional volume is defined around a reference position of the camera device, wherein the three-dimensional volume has a two-dimensional polygonal cross-section within the plane of movement. The three dimensional volume is then fitted to the depth values to determine dimensions for the polygonal cross-section. These dimensions then provide an estimate for the shape of the enclosed space.

Estimating dimensions for an enclosed space using a multi-directional camera

Certain examples described herein enable semantically-labelled representations of a three-dimensional (3D) space to be generated from video data. In described examples, a 3D representation is a surface element or 'surfel' representation, where the geometry of the space is modelled using a plurality of surfaces that are defined within a 3D co-ordinate system. Object-label probability values for spatial elements of frames of video data may be determined using a two-dimensional image classifier. Surface elements that correspond to the spatial elements are identified based on a projection of the surface element representation using an estimated pose for a frame. Object-label probability values for the surface elements are then updated based on the object-label probability values for corresponding spatial elements. This results in a semantically-labelled 3D surface element representation of objects present in the video data. This data enables computer vision and/or robotic applications to make better use of the 3D representation.

Detecting objects in video data

Examples described herein relate to mapping a space using a multi-directional camera. This mapping may be performed with a robotic device comprising a monocular multi-directional camera device and at least one movement actuator. The mapping may generate an occupancy map to determine navigable portions of the space. A robotic device movement around a point in a plane of movement may be instructed using the at least one movement actuator. Using the monocular multi-directional camera device, a sequence of images are obtained () at different angular positions during the instructed movement. Pose data is determined () from the sequence of images. The pose data is determined using features detected within the sequence of images. Depth values are then estimated () by evaluating a volumetric function of the sequence of images and the pose data. The depth values are processed () to populate the occupancy map for the space.

Mapping a space using a multi-directional camera

Certain examples described herein relate to modelling a three-dimensional space. Image data from at least one capture device is used to generate a three-dimensional model of the three-dimensional space. In certain examples, the three-dimensional model is segmented into active and inactive portions based on at least one model property. The examples are configured to use the active portions to update the three-dimensional model over time. Registration is also performed to align active portions of the three-dimensional model with inactive portions of the three-dimensional model over time. The registration aligns active portions of the three-dimensional model generated following an observation of a region of the three-dimensional space with inactive portions of the model generated following at least one previous observation of said region.

Growing community of inventors

London, United Kingdom

Andrew Davison