Apple Inc.

Gregory Lutter

Anshu Kameswar Chimalamarri

Rahul Nair

Devin William Chalmers

Bryce L Schmidtchen

Luis R Deliz Centeno

Jonathan C Moisant-Thompson

Stephen Olivier Lemay

Giancarlo Yerkes

Allison W Dryer

Benjamin S Phipps

Thomas G Salter

William D Lindmeier

Thomas J Moore

Ioana Negoita

Brian Warren Temple

Paulo R Jansen Dos Reis

Yutaka Yokokawa

Peter Burgner

Nahckjoon Kim

Jessica Trinh

Ashwin K Vijay

Guilherme Klink

Trevor J McIntyre

Brandon K Schmuck

Andrew Muehlhausen

Trevor J Mcintyre

A head-mounted device may be used to control one or more external electronic devices. Gaze input and camera images may be used to determine a point of gaze relative to a display for an external electronic device. The external electronic device may receive information regarding the user's gaze input from the head-mounted device and may highlight one out of multiple user interface elements that is targeted by the gaze input. The head-mounted device may receive input such as keystroke information from an accessory device and relay the input to an external electronic device that is being viewed. The head-mounted device may receive a display configuration request, determine layout information for displays in the physical environment of the head-mounted device, and transmit the layout information to an external device associated with the displays.

Controlling devices using a wearable device

In some embodiments, the present disclosure includes techniques and user interfaces for interacting with graphical user interfaces using gaze. In some embodiments, the present disclosure includes techniques and user interfaces for repositioning virtual objects. In some embodiments, the present disclosure includes techniques and user interfaces for transitioning modes of a camera capture user interface.

Devices, methods, for interacting with graphical user interfaces

Various implementations disclosed herein include devices, systems, and methods for using a gaze vector and head pose information to effectuate a user interaction with a virtual object. In some implementations, a device includes a sensor for sensing a head pose of a user, a display, one or more processors, and a memory. In various implementations, a method includes displaying a set of virtual objects. Based on a gaze vector, it is determined that a gaze of the user is directed to a first virtual object of the set of virtual objects. A head pose value corresponding to the head pose of the user is obtained. An action relative to the first virtual object is performed based on the head pose value satisfying a head pose criterion.

Gaze and head pose interaction

A method includes displaying a plurality of visual elements. The method includes determining, based on respective characteristic values of the plurality of visual elements, an expected gaze target that indicates a first display region where a user of the device intends to gaze while the plurality of visual elements is being displayed. The method includes obtaining, via the image sensor, an image that includes a set of pixels corresponding to a pupil of the user of the device. The method includes determining, by a gaze tracker, based on the set of pixels corresponding to the pupil, a measured gaze target that indicates a second display region where the user is measuredly gazing. The method includes adjusting a calibration parameter of the gaze tracker based on a difference between the first display region indicated by the expected gaze target and the second display region indicated by the measured gaze target.

Calibrating a gaze tracker

Various implementations disclosed herein include devices, systems, and methods for modifying display of virtual content. In some implementations, the method is performed by an electronic device including a non-transitory memory, one or more processors, a display, an eye tracker and a movement sensor. In some implementations, the method includes displaying, via the display, virtual content. In some implementations, the method includes detecting, via the eye tracker, a movement of a gaze in a first direction. In some implementations, the method includes detecting, via the movement sensor, a movement of a head in a second direction while detecting the movement of the gaze in the first direction. In some implementations, the method includes, in accordance with a determination that the movement of the gaze in the first direction and the movement of the head in the second direction are corresponding opposing movements, modifying a display characteristic of the virtual content.

Modifying display of content based on gaze and head movement

In accordance with some embodiments, exemplary processes for changing (e.g., non-linearly) the size of a virtual object are described.

Techniques for resizing virtual objects

A head-mounted device may determine contextual information by analyzing sensor data. The head-mounted device may use computer vision analysis to determine contextual information from images of the physical environment around the head-mounted device. Instead or in addition, the head-mounted device may determine contextual information by receiving state information directly from external equipment within the physical environment. Based on the received state information, the head-mounted device may display content, play audio, change a device setting on the head-mounted device, the external equipment, and/or additional external equipment, and/or may open an application. The head-mounted device may receive the state information in accordance with identifying the external equipment in images of the physical environment. The head-mounted device may receive the state information and then obtain and analyze sensor data to determine contextual information in response to the received state information being insufficient.

Displaying content based on state information

One or more techniques for managing virtual objects between one or more displays are described. In accordance with some embodiments, exemplary techniques for displaying a virtual object are described.

Managing devices having additive displays

A head-mounted device may use head pose changes for user input. In particular, a display in the head-mounted device may display a slider with an indicator. The slider may be a visual representation of a scalar quantity of a device setting such as volume or brightness. Based on head pose changes, the scalar quantity of the device setting and the position of the indicator on the slider may be updated. The direction of a head movement may correspond to the direction of movement of the indicator in the slider. The scalar quantity of a device setting may only be updated when gaze input from a user targets the slider. The slider may be displayed in response to gaze input targeting an icon associated with the slider.

Controlling a device setting using head pose

A method is performed at an electronic device with one or more processors, a non-transitory memory, an eye tracker, and a display. The eye tracker receives eye tracking data associated with one or more eyes of a user of the electronic device. The method includes displaying, on the display, a first user interface (UI) element that is associated with a first selection region and a second selection region. While displaying the first UI element, the method includes determining, based on the eye tracking data, that a first targeting criterion is satisfied with respect to the first selection region, and determining, based on the eye tracking data, that a second targeting criterion is satisfied with respect to the second selection region. The method includes selecting the first UI element based at least in part on determining that the first targeting criterion is satisfied and the second targeting criterion is satisfied.

Eye tracking based selection of a user interface (UI) element based on targeting criteria

In one implementation, a method is performed for selecting a UI element with eye tracking-based attention accumulators. The method includes: while a first UI element is currently selected, detecting a gaze direction directed to a second UI element; in response to detecting the gaze direction directed to the second UI element, decreasing a first attention accumulator value associated with the first UI element and increasing a second attention accumulator value associated with the second UI element; in accordance with a determination that the second attention accumulator value exceeds the first attention accumulator value, deselecting the first UI element and selecting the second UI element; and in accordance with a determination that the second attention accumulator value does not exceed the first attention accumulator value, maintaining selection of the first UI element.

Growing community of inventors

Boulder Creek, CA, United States of America

Gregory Lutter