Magic Leap, Inc.

Amazon Technologies, Inc.

Ashwin Swaminathan

Ali Shahrokni

Miguel Andres Granados Velasquez

Anush Mohan

Xuan Zhao

Eran Guendelman

Prateek Singhal

Christian Ivan Robert Moore

Martin Georg Zahnert

Dominik Michael Kasper

Joao Antonio Pereira Faro

Daniel Olshansky

Rafael Domingos Torres

Joel David Holder

Javier Victorio Gomez Gonzalez

Mukta Prasad

Evan Gregory Levine

Samuel A Miller

Sunil Sharadchandra Hadap

Robert Blake Taylor

Jehangir Tajik

Lomesh Agarwal

Hélder Toshiro Suzuki

Siddharth Choudhary

Keng-Sheng Lin

Gitika Karumuri

Jeremy Dwayne Miranda

Omid Ghasemalizadeh

Frank Thomas Steinbruecker

Benjamin Langmann

Daniel E Koppel

Igor Napolskikh

Sergiu Sima

Yuelong Li

Christoph Tobler

Yu-Hsiang Huang

Manel Quim Sanchez Nicuesa

Hiral Honar Barot

Lipu Zhou

Blazej Marek Czuprynski

Lei Huang

Jean-Yves Bouguet

Etienne Gregoire Grossmann

Michael Harold Liebenow

Dennis William Strelow

Divya Sharma

Nicholas Atkinson Kramer

Ben Weisbih

Moshe Bouhnik

Sen Lin

Danying Hu

Miriam Bellver Bueno

Xin Shen

Amogh Gupta

Yu-Tseh Chi

Adam Harmat

Kenneth William Tossell

Raghavan Manmatha

Vinayram Balakumar

Siqi Deng

Hao Yang

Ali Caner Turkmen

Kailash Budhathoki

Varad Gunjal

Christopher Michael Russell

Matthaeus Kleindessner

Hirai Honar Barot

Etienne Gregoire Grossman

Hui Ju

Konstantinos Zampogiannis

Pooja Piyush Mehta

João Antonio Pereira Faro

Manuel Quim Sanchez Nicuesa

A portable electronic system receives a set of one or more canonical maps and determines the sparse map based at least in part upon one or more anchors pertaining to the physical environment. The sparse map is localized to at least one canonical map in the set of one or more canonical maps, and a new canonical map is created at least by merging sparse map data of the sparse map into the at least one canonical map. The set of one or more canonical maps may be determined from a universe of canonical maps comprising a plurality of canonical maps by applying a hierarchical filtering scheme to the universe. The sparse map may be localized to the at least one canonical map at least by splitting the sparse map into a plurality of connected components and by one or more merger operations.

Cross reality system with WiFi/GPS based map merge

A cross reality system enables any of multiple devices to efficiently and accurately access previously persisted maps, even maps of very large environments, and render virtual content specified in relation to those maps. The cross reality system may quickly process a batch of images acquired with a portable device to determine whether there is sufficient consistency across the batch in the computed localization. Processing on at least one image from the batch may determine a rough localization of the device to the map. This rough localization result may be used in a refined localization process for the image for which it was generated. The rough localization result may also be selectively propagated to a refined localization process for other images in the batch, enabling rough localization processing to be skipped for the other images.

Cross reality system with fast localization

A cross reality system enables any of multiple devices to efficiently and accurately access previously persisted maps of very large scale environments and render virtual content specified in relation to those maps. The cross reality system may build a persisted map, which may be in canonical form, by merging tracking maps from the multiple devices. A map merge process determines mergibility of a tracking map with a canonical map and merges a tracking map with a canonical map in accordance with mergibility criteria, such as, when a gravity direction of the tracking map aligns with a gravity direction of the canonical map. Refraining from merging maps if the orientation of the tracking map with respect to gravity is not preserved avoids distortions in persisted maps and results in multiple devices, which may use the maps to determine their locations, to present more realistic and immersive experiences for their users.

Cross reality system with accurate shared maps

A cross reality system enables any of multiple devices to efficiently access previously stored maps. Both stored maps and tracking maps used by portable devices may have any of multiple types of location metadata associated with them. The location metadata may be used to select a set of candidate maps for operations, such as localization or map merge, that involve finding a match between a location defined by location information from a portable device and any of a number of previously stored maps. The types of location metadata may prioritized for use in selecting the subset. To aid in selection of candidate maps, a universe of stored maps may be indexed based on geo-location information. A cross reality platform may update that index as it interacts with devices that supply geo-location information in connection with location information and may propagate that geo-location information to devices that do not supply it.

Cross reality system with prioritization of geolocation information for localization

Examples of the disclosure describe systems and methods for presenting virtual content on a wearable head device. In some embodiments, a state of a wearable head device is determined by minimizing a total error based on a reduced weight associated with a reprojection error. A view reflecting the determined state of the wearable head device is presented via a display of the wearable head device. In some embodiments, a wearable head device calculates a preintegration term based on the image data received via a sensor of the wearable head device and the inertial data received via a first IMU and a second IMU of the wearable head device. The wearable head device estimates a position of the device based on the preintegration term, and the wearable head device presents the virtual content based on the position of the device.

Dual IMU SLAM

The present disclosure generally relates to systems and methods for searching media content. In some implementation examples, a search system receives an input query, generates a query embedding of the input query, and generates a bias mitigation transformation associated with a sensitive attribute. Based on the query embedding and the bias mitigation transformation, the search system generates a transformed query embedding that suppresses at least a portion of the query embedding related to the sensitive attribute. Using the transformed query embedding, the search system executes a similarity search in a media embedding model to identify one or more media embeddings that are similar to the transformed query embedding and transmits the one or more media embeddings.

Mitigating bias in multimodal models via query transformation

To determine the head pose of a user, a head-mounted display system having an imaging device can obtain a current image of a real-world environment, with points corresponding to salient points which will be used to determine the head pose. The salient points are patch-based and include: a first salient point being projected onto the current image from a previous image, and with a second salient point included in the current image being extracted from the current image. Each salient point is subsequently matched with real-world points based on descriptor-based map information indicating locations of salient points in the real-world environment. The orientation of the imaging devices is determined based on the matching and based on the relative positions of the salient points in the view captured in the current image. The orientation may be used to extrapolate the head pose of the wearer of the head-mounted display system.

Enhanced pose determination for display device

A first region within image data corresponding to a first portion of a user is determined, and a classifier associated with the first portion of the user is determined. A second region within the image data that includes the first region is determined. Within the second region, first pixels corresponding to the first portion of the user, second pixels corresponding to a second portion of the user adjoined to the first portion of the user, and third pixels corresponding to an environment of the user are identified. Based on identifying the second pixels, the classifier associated with the first portion of the user is confirmed. Upon the confirmation an item is rendered on the first portion of the user.

Reducing false positive detections based on classification and segmentation cues

The present disclosure describes a system and method for implementing a virtual shoe try-on feature. The method includes predicting, using a machine learning model, a first plurality of pixels including pixels of an image corresponding to a foot in the image and adding a first virtual cylindrical object to a three-dimensional model of a shoe such that a portion of the first virtual cylindrical object is positioned within the three-dimensional model of the shoe. The method also include determining a set of pixels including pixels of the first plurality of pixels that intersect with the first virtual cylindrical object and removing, from the three-dimensional model of the shoe, a first portion of the three-dimensional model of the shoe that intersects with the set of pixels.

Virtual shoe try-on

A method of efficiently and accurately computing a pose of an image with respect to other image information. The image may be acquired with a camera on a portable device and the other information may be a map, such that the computation of pose localizes the device relative to the map. Such a technique may be applied in a cross reality system to enable devices to efficiently and accurately access previously persisted maps. Localizing with respect to a map may enable multiple cross reality devices to render virtual content at locations specified in relation to those maps, providing an enhanced experience for uses of the system. The method may be used in other devices and for other purposes, such as for navigation of autonomous vehicles.

Efficient localization based on multiple feature types

Methods and apparatus for providing a representation of an environment, for example, in an XR system, and any suitable computer vision and robotics applications. A representation of an environment may include one or more planar features. The representation of the environment may be provided by jointly optimizing plane parameters of the planar features and sensor poses that the planar features are observed at. The joint optimization may be based on a reduced matrix and a reduced residual vector in lieu of the Jacobian matrix and the original residual vector.

Computationally efficient method for computing a composite representation of a 3D environment

The present disclosure describes a system and method for implementing a virtual shoe try-on feature. The method includes receiving, from a database, a three-dimensional model of a shoe with a first annotated point on the three-dimensional model of the shoe, a second annotated point on the three-dimensional model of the shoe, and a third annotated point on the three-dimensional model of the shoe. The method also includes predicting, using a machine learning model and based on an image of a foot, one or more features of the foot, generating, based on the first annotated point, the second annotated point, the third annotated point, and the one or more features, a two-dimensional model of the shoe using the three-dimensional model of the shoe, and superimposing the two-dimensional model of the shoe onto the image of the foot.

A system form localizing an electronic device with dynamic buffering identifies, from the buffer, a first set of features that is extracted from a first image captured by the electronic device and receives, at the system, a second set of features that is extracted from a second image captured by the electronic device. The system further determines a first characteristic for the first set of features and a second characteristic for the second set of features and determines whether a triggering condition for dynamically changing a size of the buffer is satisfied based at least in part upon the first characteristic for the first set of features and the second characteristic for the second set of features.

Cross reality system with buffering for localization accuracy

A cross reality system enables any of multiple devices to efficiently and accurately access previously stored maps and render virtual content specified in relation to those maps. The cross reality system may include a cloud-based localization service that responds to requests from devices to localize with respect to a stored map. The request may include one or more sets of feature descriptors extracted from an image of the physical world around the device. Those features may be posed relative to a coordinate frame used by the local device. The localization service may identify one or more stored maps with a matching set of features. Based on a transformation required to align the features from the device with the matching set of features, the localization service may compute and return to the device a transformation to relate its local coordinate frame to a coordinate frame of the stored map.

Cross reality system with localization service

A cross reality system that provides an immersive user experience by storing persistent spatial information about the physical world that one or multiple user devices can access to determine position within the physical world and that applications can access to specify the position of virtual objects within the physical world. Persistent spatial information enables users to have a shared virtual, as well as physical, experience when interacting with the cross reality system. Further, persistent spatial information may be used in maps of the physical world, enabling one or multiple devices to access and localize into previously stored maps, reducing the need to map a physical space before using the cross reality system in it. Persistent spatial information may be stored as persistent coordinate frames, which may include a transformation relative to a reference orientation and information derived from images in a location corresponding to the persistent coordinate frame.

Cross reality system

Growing community of inventors

Dublin, CA, United States of America

Ashwin Swaminathan