Apple Inc.

Angela Blechschmidt

Daniel Ulbricht

Mohammad Haris Baig

Eshan Verma

Tanmay Batra

Chen-Yu Lee

Amit Kumar K C

Alexander S Polichroniadis

Amit Kumar Kc

Payal Jotwani

Mark E Drummond

Bo Morgan

Omar Elafifi

Mingshan Wang

Deep Chakraborty

In one implementation, a method of generating a scene graph includes generating the scene graph using an end-to-end scene graph generator comprising an integrated neural network. For example, in various implementations, the method includes obtaining an image representing a plurality of objects. The method includes determining a relationship vector indicating spatial relationships between a particular object of the plurality of objects and others of the plurality of objects. The method includes determining, based on the relationship vector, an object type of the particular object.

End-to-end scene graph generation

Implementations disclosed herein provide systems and methods that determine relationships between objects based on an original semantic mesh of vertices and faces that represent the 3D geometry of a physical environment. Such an original semantic mesh may be generated and used to provide input to a machine learning model that estimates relationships between the objects in the physical environment. For example, the machine learning model may output a graph of nodes and edges indicating that a vase is on top of a table or that a particular instance of a vase, V1, is on top of a particular instance of a table, T1.

Object relationship estimation from a 3D semantic mesh

Various implementations disclosed herein include devices, systems, and methods that uses object relationships represented in the scene graph to adjust the position of objects. For example, an example process may include obtaining a three-dimensional (3D) representation of a physical environment that was generated based on sensor data obtained during a scanning process, detecting positions of a set of objects in the physical environment based on the 3D representation, generating a scene graph for the 3D representation of the physical environment based on the detected positions of the set of objects, wherein the scene graph represents the set of objects and relationships between the objects, and determining a refined 3D representation of the physical environment by refining the position of at least one object in the set of objects based on the scene graph and an alignment rule associated with a relationship in the scene graph.

Object correction using scene graphs

An exemplary process obtains sensor data for a physical environment, generates a local scene graph for the physical environment based on the sensor data, wherein the local scene graph represents a set of objects and relationships between the objects, matches the local scene graph with a principal scene graph of a set of principal scene graphs, and executes one or more scripted actions involving the objects based on a narrative associated with the matched principal scene graph. In some implementations, the set of principal scene graphs is generated by generating local scene graphs for a plurality of environments, and generating individual scene graphs each representative of local scene graphs.

Scripted interactive scenarios based on scene graph matching

In one implementation, a method of generating an object model is performed at a device including one or more processors and non-transitory memory. The method includes obtaining a point cloud including a plurality of points, wherein each of the plurality of points is associated with a plurality of sets of coordinates in a three-dimensional space at a respective plurality of time periods. The method includes generating an object model including a plurality of vertices corresponding to the plurality of points and a plurality of edges between respective pairs of the plurality of vertices. The method includes generating, based on the sets of coordinates of a particular point, a characterization vector of a particular vertex corresponding to the particular point, wherein the characterization vector includes a joint label and joint data with respect to one or more of the plurality of edges of the particular vertex.

Modeling object model vertices as joints based on temporally variable point clouds

In one implementation, a method of achieving an objective using a scene graph includes identifying a set of transition couples of a plurality of transition couples between sets of spatial relationships of the scene graph, wherein each spatial relationship of a respective first set of spatial relationships indicated by each transition couple of the set of transition couples is included in an initial set of spatial relationships or an respective second set of spatial relationships indicated by a previous transition couple and wherein a particular spatial relationship is included in a respective second set of spatial relationships of a last transition couple of the set of transition couples.

Scene graph assisted navigation

Implementations disclosed herein provide systems and methods that match a current relationship model associated with a user's current environment to a prior relationship model for a prior environment to determine that the user is in the same environment. The current relationship model is compared with the prior relationship model based on matching characteristics of the current relationship model with characteristics of the prior relationship model.

Scene graph-based scene re-identification

In one implementation, a method of generating a plane hypothesis is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes obtaining an image of a scene including a plurality of pixels. The method includes obtaining a plurality of points of a point cloud based on the image of the scene. The method includes obtaining an object classification set based on the image of the scene. Each element of the object classification set includes a plurality of pixels respectively associated with a corresponding object in the scene. The method includes detecting a plane within the scene by identifying a subset of the plurality of points of the point cloud that correspond to a particular element of the object classification set.

Plane detection using semantic segmentation

In one implementation, a method of including a person in a CGR experience or excluding the person from the CGR experience is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes, while presenting a CGR experience, capturing an image of scene; detecting, in the image of the scene, a person; and determining an identity of the person. The method includes determining, based on the identity of the person, whether to include the person in the CGR experience or exclude the person from the CGR experience. The method includes presenting the CGR experience based on the determination.

Identity-based inclusion/exclusion in a computer-generated reality experience

Various implementations disclosed herein include devices, systems, and methods that enable faster and more efficient real-time physical object recognition, information retrieval, and updating of a CGR environment. In some implementations, the CGR environment is provided at a first device based on a classification of the physical object, image or video data including the physical object is transmitted by the first device to a second device, and the CGR environment is updated by the first device based on a response associated with the physical object received from the second device.

Visual search refinement for computer generated rendering environments

Implementations disclosed herein provide systems and methods that match a current scene graph associated with a user's current environment to a prior scene graph for a prior environment to determine that the user is in the same environment. The current scene graph is compared with the prior scene graph based on matching the objects within the current scene graph with objects within the prior scene graph.

A machine learning model is trained and used to perform a computer vision task such as semantic segmentation or normal direction prediction. The model uses a current image of a physical setting and input generated from three dimensional (3D) anchor points that store information determined from prior assessments of the physical setting. The 3D anchor points store previously-determined computer vision task information for the physical setting for particular 3D points locations in a 3D worlds space, e.g., an x, y, z coordinate system that is independent of image capture device pose. For example, 3D anchor points may store previously-determined semantic labels or normal directions for 3D points identified by simultaneous localization and mapping (SLAM) processes. The 3D anchor points are stored and used to generate input for the machine model as the model continues to reason about future images of the physical setting.

Temporal and geometric consistency in physical setting understanding

A machine learning (ML) model is trained and used to produce a probability distribution associated with a computer vision task. The ML model uses a prior probability distribution associated with a particular image capture condition determined based on sensor data. For example, given that an image was captured by an image capture device at a particular height above the floor and angle relative to the vertical world axis, a prior probability distribution for that particular image capture device condition can be used in performing a computer vision task on the image. Accordingly, the machine learning model is given the image as input as well as the prior probability distribution for the particular image capture device condition. The use of the prior probability distribution can improve the accuracy, efficiency, or effectiveness of the ML learning model for the computer vison task.

Growing community of inventors

San Jose, CA, United States of America

Angela Blechschmidt