Apple Inc.

Ranjit Desai

Maneli Noorkami

Michael J Rockwell

Perry A Caro

Joel N Kerr

Venu Madhav Duggineni

Afshin Taghavi Nasrabadi

Gurjeet S Saund

Robert Seon Wai Lee

Kaushik Raghunath

Rahul Nair

Aleksandr M Movshovich

Adrian Peter Lindberg

Avi Bar-zeev

Alexsandr M Movshovich

Russell L Jones

Saul H Weiss

Manjunath M Venkatesh

Michael C Friedman

Borna Ghavam

Arun Kannan

Vinay Palakkode

Alvin Li Lai

A head-mounted device is provided that includes one or more cameras configured to acquire a raw video feed and one or more displays configured to present a passthrough video feed to a user. Generation of the passthrough video feed can involve processing the raw video feed using an image signal processor and auxiliary compute blocks. One or more of the auxiliary compute blocks can be bypassed in response to detecting one or more failures associated with the auxiliary compute blocks. Configured and operated in this way, the head-mounted device can fall back to a more reliable passthrough video feed without having to power cycle the head-mounted device when a failure occurs.

Head-mounted electronic device with reliable passthrough video fallback capability

Various implementations provide a method for determining how a second user prefers to be depicted/augmented in a first user's view of a multi-user environment in a privacy preserving way. For example, a method may include determining that a physical environment includes a second device, where a second user associated with the second device is to be depicted in a view of a three-dimensional (3D) environment. The method may further include determining position data indicative of a location of the second device relative to the first device. The method may further include sending the position data indicative of the location of the second device relative to the first device, to an information system (e.g., a user preference system). The method may further include receiving a user preference setting associated with the second user for depicting or augmenting the second user in the 3D environment from the information system.

Privately sharing user preferences

Implementations of the subject technology provide systems and methods for recording an extended reality experience in a way that allows the experience to be played back at a later time from a different viewpoint or perspective. This allows computer-generated content that was rendered for display to a user during the recording, to be re-rendered during playback at the correct time and location in the recording, but from a different perspective. In order to facilitate this type of viewer-centric playback, the recording includes a computer-generated content track that references resources for re-rendering the computer-generated content at each point in time in the recording.

Extended reality recorder

Techniques are disclosed relating to encoding recorded content for distribution to other computing devices. In some embodiments, a first computing device creates recorded content for transmission to a second computing device configured to present the recorded content. To encode the recorded content, the first computing device detects, via a network interface of the first computing device, one or more computing nodes available to encode the recorded content in one or more formats supported by the second computing device. The first computing device offloads the recorded content via the network interface to the one or more computing nodes for encoding in the one or more formats. In some embodiments, the second computing device receives a request from a user to stream content recorded by a first computing device and requests the content in a first format being encoded by a computing node assisting the first computing device.

Methods and systems for encoding and decoding data

Techniques are disclosed relating to display devices. In some embodiments, a display device includes a display system configured to display three-dimensional content to a user. The display device is configured to discover, via a network interface, one or more compute nodes operable to facilitate rendering the three-dimensional content and receive information identifying abilities of the one or more compute nodes to facilitate the rendering. Based on the received information, the display device evaluates a set of tasks to identify one or more of the tasks to offload to the one or more compute nodes for facilitating the rendering and distributes, via the network interface, the identified one or more tasks to the one or more compute nodes for processing by the one or more compute nodes.

Distributed processing in computer generated reality system

In one implementation, a method of accessing shared data among processes is performed by a device including processor(s), non-transitory memory, and an image acquisition interface. The method includes obtaining image data acquired by the image acquisition interface. The method further includes determining pose data based at least in part on inertial measurement unit (IMU) information measured by the image acquisition interface. The method also includes determining a gaze estimation based at least in part on eye tracking information obtained through the image acquisition interface. Based at least in part on characteristics of processes, the method includes determining an arrangement for the image data, the pose data, and the gaze estimation. The method additionally includes determining an access schedule for the processes based at least in part on at least one of: the arrangement, the characteristics of the processes, and hardware timing parameters associated with the device.

Method and device for process data sharing

Implementations of the subject technology provides analyzing a recording of content. The subject technology generates metadata information based at least in part on the analyzing. The subject technology identifies, based at least in part on at least one of a user preference or a detected event, a region of interest or an object of interest in the recording of content. Based at least in part on the identified region of interest or object of interest, the subject technology generates a modified version of the recording of content. Further, the subject technology stores the modified version of the recording of content for subsequent playback on an electronic device.

Computer-generated reality recorder

A decoding computing device determines a focus area of a view of volumetric visual content, such as a three-dimensional (3D) object or scene, and indicates the focus area in a request for compressed volumetric visual content. A server provides a depth map and an attribute atlas that includes attribute and/or texture information for the focus area that is signaled at a higher resolution than other portions of the volumetric visual content. The decoding device also applies one or more mesh simplification techniques, wherein a higher resolution mesh is used for the focus area. The textures/attributes are projected onto the mesh representations, wherein the mesh representations and projected textures are used to reconstruct the volumetric visual content. Hole filling techniques are then applied, wherein a more sophisticated hole filling technique is used in the focus area.

Immersive media content encoding and rendering

Aspects of the subject technology relate to encoding of video frames having content with a variable resolution that varies within the video frame. Aspects of the subject technology can provide an efficient encoding by using smaller macroblocks for lower resolution content within the video frame, and larger macroblocks for higher resolution content within the video frame. An encoder may be provided with resolution information for the content of the video frame, which can be used by the encoder to determine macroblock sizes, macroblock divisions, and/or prediction modes for the encoding of the video frame.

Resolution-based video encoding

Implementations of the subject technology provide determining an operating mode of an electronic device based at least in part on whether the electronic device is communicatively coupled to an associated base device. Based on the determined operating mode, the subject technology identifies a set of input modalities for initiating a recording of content within a field of view of the electronic device. The subject technology monitors sensor information generated by at least one sensor included in, or communicatively coupled to, the electronic device. Further, the subject technology initiates the recording of content within the field of view of the electronic device when the monitored sensor information indicates that at least one of the identified set of input modalities has been triggered.

Multimodal inputs for computer-generated reality

Implementations of the subject technology provides analyzing a recording of content within a field of view of a device, the analyzing including recognition of a set of objects included in the content. The subject technology identifies a subset of the set of objects that are indicated as corresponding to protected content. The subject technology generates a modified version of the recording that obfuscates or filters the subset of the set of objects. Additionally, the subject technology provides the modified version of the recording to a host application for playback.

Providing restrictions in computer-generated reality recordings

Techniques are disclosed relating to identifying connection quality for locations within an environment. A first computing device may collect environment information pertaining to a layout of an environment in which the first computing device and a second computing device are located. The first computing device may be in wireless communication with the second computing device. The first computing device may determine, based on the environment information, a location within the environment to locate the first computing device to improve the wireless communication. The first computing device may provide an indication of the location to a user of the first computing device.

Connection assessment system

A method is provided that includes receiving content data captured by a sensor and receiving a context signal representing a user context. The received content data is scaled using a trained model, wherein the context signal is an input to the trained model, and the scaled content data is provided for presentation to a user.

Energy efficient context relevant processing for content

Techniques are disclosed relating to encoding recorded content for distribution to other computing devices. In various embodiments, a first computing device records content of a physical environment in which the first computing device is located, the content being deliverable to a second computing device configured to present a corresponding environment based on the recorded content and content recorded by one or more additional computing devices. The first computing device determines a pose of the first computing device within the physical environment and encodes the pose in a manifest usable to stream the content recorded by the first computing device to the second computing device. The encoded pose is usable by the second computing device to determine whether to stream the content recorded by the first computing device.

Efficient delivery of multi-camera interactive content

Systems and methods for gaze-driven recording of video are described. Some implementations may include accessing gaze data captured using one or more gaze-tracking sensors; applying a temporal filter to the gaze data to obtain a smoothed gaze estimate; determining a region of interest based on the smoothed gaze estimate, wherein the region of interest identifies a subset of a field of view; accessing a frame of video; recording a portion of the frame associated with the region of interest as an enhanced frame of video, wherein the portion of the frame corresponds to a smaller field of view than the frame; and storing, transmitting, or displaying the enhanced frame of video.

Growing community of inventors

Cupertino, CA, United States of America

Ranjit Desai