Microsoft Technology Licensing, LLC

Google Inc.

Xirgo Technologies, LLC

Owl Cameras, Inc.

Jeffrey Neil Margolis

Alex Aben-Athar Kipman

Mark James Finocchio

Stephen G Latta

Kathryn Stone Perez

Andrew John Fuller

Ashraf A Michail

Nathan Ackerman

Martin Thomas Shetter

Marc S Oshiro

Andrew Martin Pearson

Calvin Chan

Jeffrey C Fong

Kevin Geisner

Daniel Joseph McCulloch

Barry J Corlett

Jason S Flaks

John E Elsbree

Kort Danner Sands

Rudy Jacobus Poot

Ben J Sugden

Jason Bradley Scott

Alex A Kipman

Robert L Crocco, Jr

Brian E Keane

Kyungsuk David Lee

Sheridan Martin Small

Georg Klein

Niranjan Achugundla Puttaswamy

Chris Miles

Avi Bar-Zeev

Andrew Bart Hodge

William Hong Vong

John Allen Tardif

Arthur Charles Tomlin

Relja Markovic

Brian James Mount

Charles Walter Boecker

Darren Bennett

Ryan Lucas Hastings

Bernard Charles Kress

Michael Isard

Alexandru Octavian Balan

Barry Thompson

Chuan Pu

Paul Ronald Barham

Jay F Hamlin

Kevin Woo

Sheethal Somesh Nayak

Oliver Michael Christian Williams

Douglas Kevin Service

Kathryn Stone-Perez

Tuan Wong

David Douglas Bohn

Christian Klein

Joshua Owen Miller

Adam G Poulos

Niels Van Dongen

Scott Fullam

Jonathan Steed

Mathew Julian Lamb

Laura K Massey

Ali M Vassigh

Navid Poulad

Veronica Y Law

Shao Liu

Bojana Ostojic

Gregory A Martinez

Christopher E Miles

Britta Silke Hummel

Chao X Guo

Gregory Theodore Gibson

Frederic Azera

Christen E Coomer

Bob Crocco

Brian Kramp

Matthew Calbraith Crisler

Roger Sebastian Kevin Sylvan

Mark S Newell

Rayna Demaster-Smith

Jeremy Richard Hill

Anthony R Young

Andrew Lawrence Mattingly

Arnulfo Zepeda Navratil

Kristina M Voros

Zhiheng Jia

Arjun Dayal

Erika Carlson

Will Vong

Bill Flora

Tricia Lee

Joseph Michael Downing

Ashraf Ayman Michail

Jay Hamlin

Techniques include performing a calibration of the display with respect to the eye-tracking camera after usage. The calibration involves determining a current position of a reflected pixel of the display within a field of view of the eye-tracking camera and comparing that position with a calibration position, for example a position seen at the factory. The pixel may then be shifted according to the difference between the current position and the calibration position.

Extrinsic calibration between display and eye-tracking camera in head-mounted devices

A display device includes a display substrate and a backplane substrate. The display substrate includes an array of micro-LEDs forming individual pixels. The backplane substrate includes a plurality of pixel logic hardware modules. Each pixel logic hardware module includes a local memory element configured to store a multi-bit pixel intensity value of a corresponding micro-LED for an image frame. The backplane substrate is bonded to a backside of the display substrate such that the pixel logic hardware modules are physically aligned behind the array of micro-LEDs and each pixel logic hardware module is electrically connected to a micro-LED of the corresponding pixel.

Micro-LED display

A head-mounted display device including one or more position sensors and a processor. The processor may receive a rendered image of a current frame. The processor may receive position data from the one or more position sensors and determine an updated device pose based on the position data. The processor may apply a first spatial correction to color information in pixels of the rendered image at least in part by reprojecting the rendered image based on the updated device pose. The head-mounted display device may further include a display configured to apply a second spatial correction to the color information in the pixels of the rendered image at least in part by applying wobulation to the reprojected rendered image to thereby generate a sequence of wobulated pixel subframes for the current frame. The display may display the current frame by displaying the sequence of wobulated pixel subframes.

Reprojection and wobulation at head-mounted display device

Technologies are described herein for an eye tracking that may be employed by devices and systems such as head mount display (HMD) devices. Light that is reflected from a user's eye may be specular or scattered. The specular light has an intensity or magnitude that may saturate the electronics. The presently disclosed techniques mitigate saturation by generating detected signals from an optical detector, evaluating the signal levels for the detected signal, and selectively gating the detected signals that have saturated. The remaining scattered signals can be combined to achieve a combined signal that can be converted into a digital signal without saturating the electronics, which can then be processed to form an image of the eye for identification purposes, for tracking eye movement, and for other uses. The described technologies provide a clear image without ambient light reflections or specular light interfering with the image.

Eye tracking imager with gated detectors receiving a specular reflection

The techniques disclosed herein detect sensor misalignments in a display device by the use of sensors operating under different modalities. In some configurations, a near-to-eye display device can include a number of sensors that can be used to track movement of the device relative to a surrounding environment. The device can utilize multiple sensors operating under multiple modalities. For each sensor, there is a set of intrinsic and extrinsic properties that are calibrated. The device is also configured to determine refined estimations of the intrinsic and extrinsic properties at runtime. The refined estimations of the intrinsic and extrinsic properties can then be used to derive knowledge on how the device has deformed over time. The device can then use the refined estimations of the intrinsic and extrinsic properties and/or any other resulting data that quantifies any deformations of the device to make adjustments to rendered images at runtime.

Enhanced detection of sensor misalignments by coordinating sensors utilizing multiple sensing modalities

In a system for video data capture and sharing client devices may include one or more video cameras and sensors to capture video data and to generate associated metadata. A cloud-based component may receive metadata from the client devices and requests for sharing video data captured by other client devices. Client devices with requested video data are identified by matching their provided metadata to the sharing request and by their response to an image search query for an object of interest specified in the request.

Video-based data collection, image capture and analysis configuration

Examples are disclosed that related to scanning image display systems. In one example, a scanning head-mounted display system includes a light source, a motion sensor, a scanning mirror system configured to scan light from the light source along at least one dimension to form an image, and a controller configured to control the scanning mirror system to scan the light to form the image, receive head motion data from the motion sensor, and adjust one or more of a scan rate and a phase offset between a first frame and a second frame of the image based upon the head motion data.

MEMS scanning display device

The technology provides contextual personal information by a mixed reality display device system being worn by a user. A user inputs person selection criteria, and the display system sends a request for data identifying at least one person in a location of the user who satisfy the person selection criteria to a cloud based application with access to user profile data for multiple users. Upon receiving data identifying the at least one person, the display system outputs data identifying the person if he or she is within the field of view. An identifier and a position indicator of the person in the location is output if not. Directional sensors on the display device may also be used for determining a position of the person. Cloud based executing software can identify and track the positions of people based on image and non-image data from display devices in the location.

Providing location occupancy analysis via a mixed reality device

Motion vector estimation is provided for generating and displaying images at a frame rate that is greater than a rendering frame rate. The displayed images may include late stage graphical adjustments of pre-rendered scenes that incorporate motion vector estimations. A head-mounted display (HMD) device may determine a predicted pose associated with a future position and orientation of the HMD, render a current frame based on the predicted pose, determine a set of motion vectors based on the current frame and a previous frame, generate an updated image based on the set of motion vectors and the current frame, and display the updated image on the HMD. In one embodiment, the HMD may determine an updated pose associated with the HMD subsequent to or concurrent with generating the current frame, and generate the updated image based on the updated pose and the set of motion vectors.

Frame projection for augmented reality environments

Methods for generating and displaying images associated with one or more virtual objects within an augmented reality environment at a frame rate that is greater than a rendering frame rate are described. The rendering frame rate may correspond with the minimum time to render images associated with a pose of a head-mounted display device (HMD). In some embodiments, the HMD may determine a predicted pose associated with a future position and orientation of the HMD, generate a pre-rendered image based on the predicted pose, determine an updated pose associated with the HMD subsequent to generating the pre-rendered image, generate an updated image based on the updated pose and the pre-rendered image, and display the updated image on the HMD. The updated image may be generated via a homographic transformation and/or a pixel offset adjustment of the pre-rendered image by circuitry within the display.

Reprojection OLED display for augmented reality experiences

An virtual character such as an on-screen object, an avatar, an on-screen character, or the like may be animated using a live motion of a user and a pre-recorded motion. For example, a live motion of a user may be captured and a pre-recorded motion such as a pre-recorded artist generated motion, a pre-recorded motion of the user, and/or a programmatically controlled transformation may be received. The live motion may then be applied to a first portion of an the virtual character and the pre-recorded motion may be applied to a second portion of the virtual character such that the virtual character may be animated with a combination of the live and pre-recorded motions.

Systems and methods for applying animations or motions to a character

Methods for generating and displaying images associated with one or more virtual objects within an augmented reality environment at a frame rate that is greater than a rendering frame rate are described. The rendering frame rate may correspond with the minimum time to render images associated with a pose of a head-mounted display device (HMD). In some embodiments, the HMD may determine a predicted pose associated with a future position and orientation of the HMD, generate a pre-rendered image based on the predicted pose, determine an updated pose associated with the HMD subsequent to generating the pre-rendered image, generate an updated image based on the updated pose and the pre-rendered image, and display the updated image on the HMD. The updated image may be generated via a homographic transformation and/or a pixel offset adjustment of the pre-rendered image.

Late stage reprojection

Growing community of inventors

Seattle, WA, United States of America

Jeffrey Neil Margolis