Facebook Technologies, LLC

Meta Platforms Technologies, LLC

Julian Straub

Richard Andrew Newcombe

Steven John Lovegrove

Thomas John Whelan

Yuheng Ren

Jakob Julian Engel

Michael Goesele

Xinqiao Liu

Michael Anthony Hall

Renzo De Nardi

Shobhit Verma

In one embodiment, a method includes accessing a digital image captured by a camera that is connected to a machine-detectable object, detecting a reflection of the machine-detectable object in the digital image, computing, in response to the detection, a plane that is coincident with a reflective surface associated with the reflection, determining a boundary of the reflective surface in the plane based on at least one of a plurality of cues, and storing information associated with the reflective surface, where the information includes a pose of the reflective surface and the boundary of the reflective surface in a 3D model of a physical environment, and where the information associated with the reflective surface and the 3D model are configured to be used to render a reconstruction of the physical environment.

Mirror reconstruction

The disclosed computer-implemented method may include receiving, from devices in an environment, real-time data associated with the environment and determining, from the real-time data, current object data for the environment. The current object data may include both state data and relationship data for objects in the environment. The method may also include determining object deltas between the current object data and prior object data from an event graph. The prior object data may include prior state data and prior relationship data for the objects. The method may include detecting an unknown state for one of the objects, inferring a state for the object based on the event graph, and updating the event graph based on the object deltas and the inferred state. The method may further include sending updated event graph data to the devices. Various other methods, systems, and computer-readable media are also disclosed.

Augmented reality mapping systems and related methods

The disclosed computer-implemented method may include determining a local position and a local orientation of a local device in an environment and receiving, by the local device and from a mapping system, object data for objects within the environment. The object data may include position data and orientation data for the objects and relationship data between the objects. The method may also include deriving, based on the object data received from the mapping system, and the local position and orientation of the local device, a contextual rendering of the objects that provides contextual data that modifies a user's view of the environment. The method may include displaying, using the local device, the contextual rendering of at least one of the plurality of objects to modify the user's view of the environment. Various other methods, systems, and computer-readable media are also disclosed.

Systems and methods for rendering augmented reality mapping data

The disclosed computer-implemented method may include receiving, from devices in an environment, real-time data associated with the environment. The method may also include determining, from the real-time data, current mapping and object data. The current mapping data may include coordinate data for the environment and the current object data may include both state data and relationship data for objects in the environment. The method may also include determining mapping deltas between the current mapping data and baseline map data and determining object deltas between the current object data and an event graph. The event graph may include prior state data and prior relationship data for objects. The method may also include updating the baseline map data and the event graph based on the deltas and sending updated baseline map data and event graph data to the devices. Various other methods, systems, and computer-readable media are also disclosed.

A sensor apparatus may include an array of protruding members that each extend outward in a different radial direction from a central axis, each protruding member including a projector that projects structured light into a local environment, one or more cameras that capture reflections of the structured light from the local environment, and another camera that captures visible-spectrum light from the local environment. The sensor apparatus may also include one or more localization devices for determining a location of the sensor apparatus. Various other apparatuses, systems, and methods are also disclosed.

Apparatus, system, and method for mapping a 3D environment

An imaging device operates as an event camera. The device includes an event sensor and a controller. The sensor comprises a plurality of photodiodes that asynchronously output data values corresponding to relative intensity changes within a local area. The controller populates an event matrix based in part on data values asynchronously received from the sensor and positions of photodiodes associated with the received data values over a first time period. The controller populates a change matrix based in part on a threshold intensity value and the photodiodes associated with the received data values over the first time period, and generates an image for the first time period using the event matrix and the change matrix.

Growing community of inventors

Redmond, WA, United States of America

Julian Straub