Apple Inc.

Metaio Gmbh

Other

Siemens Aktiengesellschaft

Peter Meier

Daniel R Kurz

Stefan Misslinger

Lejing Wang

Sebastian Lieberknecht

Ian M Richter

Frank Angermann

Stefan Holzer

Mohamed Selim Ben Himane

Sawyer Isaac Cohen

Benjamin B Lyon

Daniel Ulbricht

Mark Rober

Michael Kuhn

Sebastian Knorr

Holly Gerhard

Tobias Holl

Jeffrey M Riepling

Omar Elafifi

Jean-Daniel E Nahmias

Mark E Drummond

Michael J Rockwell

Siva Chandra Mouli Sivapurapu

Bo Morgan

Cameron J Dunn

Brian Chris Clark

Anton Fedosov

John Christopher Russell

Ben Blachnitzky

Selim Ben Himane

Patrick Steven Piemonte

Tanmay Batra

Daniel De Rocha Rosario

Jason D Gosnell

Mehdi Hamadou

Norbert Stoeffler

Christian Lipski

Dirk Jahn

Sebastian Bernhard Knorr

Selim BenHimane

Thomas Alt

Mohamed Selim Benhimane

Thomas Severin

Masahiro Ookura

Tobias Eble

Katharina Pentenrieder

Jan-Friso Evers-Senne

Jan-Michael Frahm

Juri Platonov

Didier Stricker

Abhishek Sharma

Jens Weidenhausen

Marion Langer

Various implementations disclosed herein include devices, systems, and methods that detect surfaces and reflections in such surfaces. Some implementations involve providing a CGR environment that includes virtual content that replaces the appearance of a user or the user's device in a mirror or other surface providing a reflection. For example, a CGR environment may be modified to include a reflection of the user that does not include the device that the user is holding or wearing. In another example, the CGR environment is modified so that virtual content, such as a newer version of the electronic device or a virtual wand, replaces the electronic device in the reflection. In another example, the CGR environment is modified so that virtual content, such as a user avatar, replaces the user in the reflection.

Image-based detection of surfaces that provide specular reflections and reflection modification

An exemplary process for identifying a type of a physical environment amongst a plurality of types of physical environments is provided. The process includes obtaining, using the one or more cameras, image data corresponding to a physical environment. The process further includes identifying at least one portion of an entity in the physical environment based on the image data; determining, based on the identified at least one portion of the entity, whether the entity is an entity of a first type; determining a type of the physical environment if the entity is an entity of the first type; and presenting one or more virtual objects and a representation of the entity.

Scene classification

In one implementation, a method includes: obtaining a user input to view SR content associated with video content; if the video content includes a first scene when the user input was detected: obtaining first SR content for a first time period of the video content associated with the first scene; obtaining a task associated with the first scene; and causing presentation of the first SR content and a first indication of the task associated with the first scene; and if the video content includes a second scene when the user input was detected: obtaining second SR content for a second time period of the video content associated with the second scene; obtaining a task associated with the second scene; and causing presentation of the second SR content and a second indication of the task associated with the second scene.

Method and device for presenting synthesized reality companion content

In various implementations, a device surveys a scene and presents, within the scene, a extended reality (XR) environment including one or more assets that evolve over time (e.g., change location or age). Modeling such an XR environment at various timescales can be computationally intensive, particularly when modeling the XR environment over larger timescales. Accordingly, in various implementations, different models are used to determine the environment state of the XR environment when presenting the XR environment at different timescales.

Computationally efficient model selection

A method of displaying virtual information in a view of a real environment comprising the following steps: providing the system relative to at least one part of the real environment and providing accuracy information of the current pose, providing multiple pieces of virtual information, and assigning a respective one of the pieces of virtual information to one of different parameters indicative of different pose accuracy information, and displaying at least one of the pieces of virtual information in the view of the real environment according to the accuracy information of the current pose in relation to the assigned parameter.

Method of displaying virtual information in a view of a real environment

A method of projecting digital information on a real object in a real environment includes the steps of projecting digital information on a real object or part of a real object with a visible light projector, capturing at least one image of the real object with the projected digital information using a camera, providing a depth sensor registered with the camera, the depth sensor capturing depth data of the real object or part of the real object, and calculating a spatial transformation between the visible light projector and the real object based on the at least one image and the depth data. The invention is also concerned with a corresponding system.

Method of and system for projecting digital information on a real object in a real environment

Detecting a chewing noise from a user during a chewing session, triggering operation of a camera, obtaining image data capturing a food product, identifying the food product based on image data, determining a measurement of the chewing session, determining a volume of the food product based on the measurement of the chewing session, and determining a calorie intake based on the food product, the volume of the food product, and the measurement of the chewing session.

Augmented reality calorie counter

The invention relates to a method for ergonomically representing virtual information in a real environment, including the following steps: providing at least one view of a real environment and of a system setup for blending in virtual information for superimposing with the real environment in at least part of the view, the system setup comprising at least one display device, ascertaining a position and orientation of at least one part of the system setup relative to at least one component of the real environment, subdividing at least part of the view of the real environment into a plurality of regions comprising a first region and a second region, with objects of the real environment within the first region being placed closer to the system setup than objects of the real environment within the second region, and blending in at least one item of virtual information on the display device in at least part of the view of the real environment, considering the position and orientation of said at least one part of the system setup, wherein the virtual information is shown differently in the first region than in the second region with respect to the type of blending in the view of the real environment.

Method for representing virtual information in a real environment

A method for representing virtual information in a view of a real environment comprises providing a virtual object having a global position and orientation with respect to a geographic global coordinate system, with first pose data on the global position and orientation of the virtual object, in a database of a server, taking an image of a real environment by a mobile device and providing second pose data as to at which position and with which orientation with respect to the geographic global coordinate system the image was taken. The method further includes displaying the image on a display of the mobile device, accessing the virtual object in the database and positioning the virtual object in the image on the basis of the first and second pose data, manipulating the virtual object or adding a further virtual object, and providing the manipulated virtual object with modified first pose data or the further virtual object with third pose data in the database.

Method for representing virtual information in a view of a real environment

In some implementations at an electronic device, training a dual EDNN includes defining a data structure of attributes corresponding to defined parts of a task, processing a first instance of an input using a first EDNN to produce a first output while encoding a first set of the attributes in a first latent space, and processing a second instance of the input using a second EDNN to produce a second output while encoding attribute differences from attribute averages in a second latent space. The device then determines a second set of the attributes based on the attribute differences and the attribute averages. The device then adjusts parameters of the first and second EDNNs based on comparing the first instance of the input to the first output, the second instance of the input to the second output, and the first set of attributes to the second set of attributes.

Unsupervised annotation using dual network system with pre-defined structure

In one implementation, a method involves obtaining light intensity data from a stream of pixel events output by an event camera of a head-mounted device ('HMD'). Each pixel event is generated in response to a pixel sensor of the event camera detecting a change in light intensity that exceeds a comparator threshold. A set of optical sources disposed on a secondary device that are visible to the event camera are identified by recognizing defined illumination parameters associated with the optical sources using the light intensity data. Location data is generated for the optical sources in an HMD reference frame using the light intensity data. A correspondence between the secondary device and the HMD is determined by mapping the location data in the HMD reference frame to respective known locations of the optical sources relative to the secondary device reference frame.

AR/VR controller with event camera

In one implementation, a method includes: identifying a first plot-within a scene associated with a portion of video content; synthesizing a scene description for the scene that corresponds to a trajectory of the first plot-effectuator within a setting associated with the scene and actions performed by the first plot-effectuator; and generating a corresponding synthesized reality (SR) reconstruction of the scene by driving a first digital asset associated with the first plot-effectuator according to the scene description for the scene.

Method and device for generating a synthesized reality reconstruction of flat video content

A method of tracking a mobile device comprising at least one camera in a real environment comprises the steps of receiving image information associated with at least one image captured by the at least one camera, generating a first geometrical model of at least part of the real environment based on environmental data or mobile system state data acquired in an acquisition process by at least one sensor of a mobile system, which is different from the mobile device, and performing a tracking process based on the image information associated with the at least one image and at least partially according to the first geometrical model, wherein the tracking process determines at least one parameter of a pose of the mobile device relative to the real environment. The invention is also related to a method of generating a geometrical model of at least part of a real environment using image information from at least one camera of a mobile device.

Method of tracking a mobile device and method of generating a geometrical model of a real environment using a camera of a mobile device

Growing community of inventors

Munich, Germany

Peter Meier