Google Inc.

Massachusetts Institute of Technology

Microsoft Technology Licensing, LLC

Alex Olwal

Ruofei Du

Artem Dementyev

Richard Francis Lyon

Philip Dam Roadley-Battin

Tyler Gough

Hendrik Wagenaar

Andrew David Wilson

Ramesh Raskar

Thad Eugene Starner

Bernard Charles Kress

Hiroshi Ishii

Greg Ellis Priest-Dorman

David Kim

Sean Follmer

Kelly Elizabeth Dobson

Danhang Tang

Boxin Shi

Daniel Leithinger

Nina R Sinatra

Munehiko Sato

Franziska Schlagenhauf

Kurtis Robert Gross

Mathieu Simon Le Goc

Dmitrii Votintcev

Kevin Balke

Systems and methods are disclosed that address the need for adaptive exposure within high dynamic range (HDR) images. Solutions can leverage recent advances in the use of virtual reality (VR) headsets and Augmented Reality (AR) displays equipped with infrared (IR) eye tracking devices. A gaze vector determined by the eye tracking device identifies one or more fixation points on the image that corresponds to an area where there exists a faulty exposure. The exposure around the fixation point can be adaptively corrected using image processing techniques. Using spatial adaptive exposure, the resulting image, a type of foveated image, can be rendered on a low dynamic range (LDR) display with sufficient detail.

Foveated images with adaptive exposure

Smart devices can be configured to collect and share various forms of context data about where a user is located (e.g., location), what a user will be doing (e.g., schedule), and what a user is currently doing (e.g., activity). This context data may be combined with fingerprint data (e.g., biometrics) to help identify the fingerprint data. For example, a location of a user may help associated speech detected at that location with the user. These associations may be stored in a mapping database that can be updated over time to reduce ambiguities in identification. The mappings in the database may be used to train a machine learning model to recognize fingerprints as identities, which may be useful in applications, such as speaker identification.

Context-aided identification

The disclosed systems and method correlates user behaviors with audio processing to achieve more accurate conclusions about sounds in a user's environment. These conclusions may, in turn, be used to adjust the way a device, such as AR glasses, operate or respond to the sounds. For example, audio events determined from processing speech can be correlated with behavior events determined by sensing a user to improve a speech-to-text transcript of the speech by separating, or otherwise altering, the text in the transcript by speaker.

Response to sounds in an environment based on correlated audio and user events

Aspects of the disclosure provide a hybrid smartwatch that incorporates digital technology with an analog timepiece in a wristwatch form factor. A digital display layer of a non-emissive material is configured to present notices, data, content and other information. An analog display layer includes one or more hands of the timepiece, and overlies the digital display layer. The hands may be controlled by a processor through micro-stepper motors or other actuators. Physical motion of the hands is simultaneously coupled with arrangement of content or other elements on the display layer, which results in numerous types of hybrid visualizations. This includes temporal presentations using hourly, daily, monthly or other time scales. Shape-shifting of the watch hands between 2D and 1D arrangements can linearly focus on certain information. And various content-aware layouts can highlight, bracket, occlude or otherwise emphasize or deemphasize displayed information.

User interface visualizations in a hybrid smart watch

Systems and methods are related to tracking an attention of a user with respect to content presented on a virtual screen, detecting a defocus event associated with a first region of the content, and determining a next focus event associated with a second region of the content. The determination can be based at least in part on the defocus event and on the tracked attention of the user. The systems and methods can include generating, based on the determined next focus event, a marker for differentiating the second region of the content from a remainder of the content, and in response to detecting a refocus event associated with the virtual screen, triggering execution of the marker associated with the second region of the content.

Attention tracking to augment focus transitions

Subsurface display interfaces and associated systems and methods are provided. In some embodiments, an example method can include: dividing a graphic to be displayed into a plurality of sets of primitives, each primitive of the sets of primitives comprising a rectangle or a line; assigning the plurality of sets of primitives to display frames of a plurality of display frames; and outputting, at a display positioned under a surface and in series, the plurality of display frames.

Hidden display interfaces and associated systems and methods

Aspects of the technology provide a symbiotic graphical display on a client device such as a smartwatch. The system includes at least one emissive display element and at least one non-emissive display element. The display elements are arrayed in layers or other configurations such that content or other information is concurrently aligned across the respective display surfaces of the different elements. A first set of content is rendered using the non-emissive display element while a second set of content is rendered using the emissive display element. Depending on characteristics or features of a given content item, that item may be rendered by one or both of the display elements. Certain content may be transitioned from the emissive display element to the non-emissive display element according to a time threshold or other criteria.

Symbiotic smartwatch displays

According to an aspect, a method for distributed sound/image recognition using a wearable device includes receiving, via at least one sensor device, sensor data, and detecting, by a classifier of the wearable device, whether or not the sensor data includes an object of interest. The classifier configured to execute a first machine learning (ML) model. The method includes transmitting, via a wireless connection, the sensor data to a computing device in response to the object of interest being detected within the sensor data, where the sensor data is configured to be used by a second ML model on the computing device or a server computer for further sound/image classification.

Distributed sensor data processing using multiple classifiers on multiple devices

A method including receiving, by a companion device from an optical display, a distortion information associated with a geometric distortion associated with rendering an image on a display of the optical display, distorting, by the companion device, the image using the distortion information, preprocessing, by the companion device, the distorted image based on compression artifacts, compressing, by the companion device, the distorted image, and communicating, by the companion device, the compressed image to the optical display

Compression-aware pre-distortion of geometry and color in distributed graphics display systems

A method may receive a first audio signal data at a first device, the first audio signal data decoded using a microphone array. A method may determine that the first audio signal data includes an audio sequence relating to an information sharing request. A method may send a first signal. A method may receive a second signal via the microphone array responsive to the first signal. A method may verify that the second signal was sent from a direction of a voice associated with the audio sequence, the second signal including a second device identifier associated with a second device. A method may establish a wireless connection with the second device using the second device identifier.

Authenticating users using localization of speech or correlations in audio data

This document describes techniques directed to a scalable gesture sensor for wearable and soft electronic devices. The scalable gesture sensor is integrated into an object such as a wearable garment or a large-surface embedded system to provide a touch-sensitive surface for the object. The sensor includes a repeated localized crossover pattern formed by the same few sensor lines, resulting in the same two conductive lines having multiple crossover points across the touch-sensitive surface. The repeated crossover pattern enables detection of the occurrence and relative direction of a swipe gesture based at least on a repeated sequence of capacitance changes over a set of conductive lines in the repeated crossover pattern. Also, the scalable gesture sensor is computationally simple, uses low power, and is uniquely scalable to cover a large area with few electrodes.

Scalable gesture sensor for wearable and soft electronic devices

An augmented reality (AR) device, such as AR glasses, may include a microphone array. The sensitivity of the microphone array can be directed to a target by beamforming, which includes combining the audio of each microphone of the array in a particular way based on a location of the target. The present disclosure describes systems and methods to determine the location of the target based on a gaze of a user and beamform the audio accordingly. This eye-tracked beamforming (i.e., foveated beamforming) can be used by AR applications to enhance sounds from a gaze direction and to suppress sounds from other directions. Additionally, the gaze information can be used to help visualize the results of an AR application, such as speech-to-text.

Foveated beamforming for augmented reality devices and wearables

Systems and methods are provided in which physical objects in the ambient environment can function as user interface implements in an augmented reality environment. A physical object detected within a field of view of a camera of a computing device may be designated as a user interface implement in response to a user command. User interfaces may be attached to the designated physical object, to provide a tangible user interface implement for user interaction with the augmented reality environment.

Tangible six-degree-of-freedom interfaces for augmented reality

According to a general aspect, a method can include receiving a photo of a virtual conference participant, and a depth map based on the photo, and generating a plurality of synthesized images based on the photo. The plurality of synthesized images can have respective simulated gaze directions of the virtual conference participant. The method can also include receiving, during a virtual conference, an indication of a current gaze direction of the virtual conference participant. The method can further include animating, in a display of the virtual conference, an avatar corresponding with the virtual conference participant. The avatar can be based on the photo. Animating the avatar can be based on the photo, the depth map and at least one synthesized image of the plurality of synthesized images, the at least one synthesized image corresponding with the current gaze direction.

Avatar animation in virtual conferencing

A wearable device can include a wearable band configured to contact a user of the wearable device, an actuator, a sensor, and one or more processors in communication with the actuator and the sensor. The processors can be configured to measure a back electromotive force ('EMF') of the actuator; determine, based on the measured back EMF, data that describes a contact force between the wearable band and the user; and determine, based on the data that describes the contact force, a quality metric describing a data quality of sensor data collected by the sensor. In some embodiments, the processor(s) can determine, generate sensor output data based on the sensor data and based at least in part on the data describing the contact force between the wearable band and the user. For example, one or more machine-learned models maybe leveraged to generate sensor output data that is compensated for the wearable band being too tight or too loose.

Growing community of inventors

San Francisco, CA, United States of America

Alex Olwal