Apple Inc.

Adrian Peter Lindberg

Nathan L Fillhardt

Daniele Casaburo

Syed Mohsin Hasan

Elisabeth Lindkvist

Marcel Van Os

Billy P Chen

Christine B McGavran

Yaohua Hu

Cédric J Bray

Per Fahlberg

Ranjit Desai

Eshan Verma

Vincent P Arroyo

Justin M Strawn

Pedro Manuel Da Silva Quelhas

Amritpal Singh Saini

Kaushik Raghunath

Elisabeth Lindkvisk

Vinay Palakkode

A method is provided that includes receiving content data captured by a sensor and receiving a context signal representing a user context. The received content data is scaled using a trained model, wherein the context signal is an input to the trained model, and the scaled content data is provided for presentation to a user.

Energy efficient context relevant processing for content

To reduce this amount of bandwidth needed to share 3D map images between mobile devices, according to some embodiments, a user's mobile device (i.e., a host device) may identify its origin in a 3D map and a current virtual camera position relative to the origin based on the physical position of the mobile device. The mobile device may send both the origin and the virtual camera position to another mobile device (i.e., a client device) for use in rendering a corresponding image. Separately, the client device may download the 3D map images from a server, e.g., in preparation for a meeting. In this manner, the host device may send the origin to the client device once, as well as send a data stream of the current virtual camera position for use in accessing the corresponding 3D map images at the client device.

Cooperative augmented reality map interface

Implementations of the subject technology provide for image composition for extended reality systems. Image composition may include combining virtual content from virtual images with physical content from images captured by one or more cameras. The virtual content and the physical content can be combined to form a composite image using depth information for the virtual content and the physical content. An adjustment mask may be generated to indicate edges or boundaries between virtual and physical content at which artifact correction for the composite image can be performed.

Image composition for extended reality systems

Various implementations disclosed herein include devices, systems, and methods that create additional depth frames where a depth camera runs at a lower frame rate than a light intensity camera. Rather than upconverting the depth frames by simply repeating a previous depth camera frame, additional depth frames are created by adjusting some of the depth values of a prior frame based on the RGB camera data (e.g., by 'dragging' depths from their positions in the prior depth frame to new positions for a new frame). Specifically, a contour image is generated, and changes in the contour image are used to determine how to adjust (e.g., drag) the depth values for the additional depth frames. The contour image may be based on a mask (e.g., occlusions masks identifying where the hand occludes the virtual cube).

Depth rate up-conversion

In some implementations, a method includes: obtaining a logical representation of a fixed function node; generating, by concerted operation of the logical representation of the fixed function node and a machine learning node that interfaces with the logical representation of the fixed function node, a candidate result based on a set of image data frames; determining whether error criteria are satisfied based at least in part on a comparison between the candidate result and a predetermined result for the set of image data frames; and, in response to determining that the error criteria are satisfied, modifying at least one of: a first portion of operating parameters of the machine learning node associated with operations of the machine learning node; and a second portion of operating parameters of the machine learning node associated with interfacing operations between the machine learning node and the logical representation of the fixed function node.

End-to-end training of a machine learning node that interfaces with a fixed function node

Various implementations disclosed herein include devices, systems, and methods that create additional depth frames in the circumstance wherein a depth camera runs at a lower frame rate than a light intensity camera. Rather than upconverting the depth frames by simply repeating a previous depth camera frame, additional depth frames are created by adjusting some of the depth values of a prior frame based on the RGB camera data (e.g., by 'dragging' depths from their positions in the prior depth frame to new positions for a new frame). Specifically, a contour image (e.g., identifying interior and exterior outlines of a hand with respect to a virtual cube that the hand occludes) is generated based on a mask (e.g., occlusions masks identifying where the hand occludes the virtual cube). Changes in the contour image are used to determine how to adjust (e.g., drag) the depth values for the additional depth frames.

Various embodiments of the disclosure pertain to an augmented or virtual reality interface for interacting with maps displayed from a virtual camera perspective on a mobile device. Instead of manipulating the position of the virtual camera using a touchscreen interface, some embodiments allow a spatial location of the mobile device to control the position of the virtual camera. For example, a user can tilt the mobile device to obtain different angles of the virtual camera. As another example, the user can move the mobile device vertically to change the height of the virtual camera, e.g., a higher altitude above the ground.

Augmented reality interface for interacting with displayed maps

Some embodiments provide a mapping application that receives input to adjust an amount of tilt from which to view a 3D map for a viewing mode in which the 3D map is specified to be textured with camera captured images (CCIs) of the real world. The mapping application also adjusts a set of heights for a set of road path indicators in the 3D map based on the adjusted amount of tilt. Each road path indicator indicates a path of a road in the 3D map. The mapping application further displays the 3D map with the adjusted amount of tilt and the set of road path indicators at the adjusted set of heights in the 3D map.

Adjusting location indicator in 3D maps

For a device that runs a mapping application, a method for providing maneuver indicators along a route of a map. The maneuver indicators are arrows that identify the direction and orientation of a maneuver. A maneuver arrow may be selected and displayed differently from unselected maneuver arrows. Maneuver arrows may be selected automatically based on a user's current location. The mapping application transitions between maneuver arrows and provides an animation for the transition. Complex maneuvers may be indicated by multiple arrows, providing a more detailed guidance for a user of the mapping application.

Growing community of inventors

San Jose, CA, United States of America

Adrian Peter Lindberg