Microsoft Technology Licensing, LLC

Microsoft Corporation

Ashraf A Michail

Gerhard A Schneider

Kevin T Gallo

Joseph S Beda, Iii

Adam M Smith

Mark A Alcazar

Martin Thomas Shetter

Andrew Martin Pearson

Jeffrey Neil Margolis

John Green Bedworth

Michael George Boulton

Calvin Chan

Brendan Joseph Clark

Gregory David Swedberg

Arthur Charles Tomlin

Barry J Corlett

Dag Birger Frommhold

Georg Klein

Bede Jordan

Alex Aben-Athar Kipman

Kun Zhou

Michael Isard

Leonardo Esteban Blanco

Donald D Karlov

Nathan Ackerman

Akhilesh Kaza

Michael Kallay

Quentin Simon Charles Miller

Paul C David

Paul Ronald Barham

Christopher N Raubacher

Oreste Dorin Ungureanu

Adrian Secchia

Matthew W Calkins

Kevin Woo

Roger Sebastian Kevin Sylvan

Andrey E Arsov

Pawel Wiktor Olszta

Oliver Michael Christian Williams

Douglas Kevin Service

Yang You

Tuan Wong

Stephen G Latta

Denis Demandolx

Mark Stefan Grossman

Greg D Schechter

Jeffrey Alan Kohler

Michael John Hillberg

Zsolt Mathe

Sam J George

Robert Thomas Held

Ning Xu

Sujal S Parikh

Michael R Harsh

Matt Bronder

Eric W Richards

Andrew Zicheng Yeung

Jack Elliott

Matthew David Sandy

Matthew Calbraith Crisler

Christian Voss-Wolff

Jonathan Lyons

Sudeesh Reddy Pingili

Alan Linchuan Liu

Benjamin Markus Thaut

David Teitlebaum

Joseph P Stegman

Christian Voss-wolff

Gunjan A Shah

Shawn T Oster

Jonathan D Sheller

Brandon C Furtwangler

Mikhail Shatalin

Will Guyman

Jerome Raymond Halmans

Alan C T Liu

Minshik Park

Justin Nafziger

George Xin Gao

Michael J Hillberg

In various examples there is a method performed by a Head Mounted Display, HMD, comprising a high field rate display configured to display fields of rendered frames at a field rate. The method comprises receiving a stream of the rendered frames for display on the high field rate display, the stream of rendered frames having a frame rate. The process applies an early stage reprojection to the rendered frames of the stream of rendered frames at a rate which is lower than the field rate. The process applies a late stage reprojection to fields of the rendered frames at the field rate, wherein the early stage reprojection uses more computational resources than the late reprojection.

Reprojection for high field rate displays

A wearable device includes multiple subsystems including a processor and a memory device, multiple temperature sensors coupled to sense temperatures of the multiple subsystems, and programming, including an application, stored on the memory device for execution by the processor to perform operations. The operations include receiving temperature information from the multiple temperature sensors corresponding to temperatures associated with the multiple subsystems, processing the temperature information to identify a first subsystem of the multiple subsystems, and providing a notification to the application executing on the processor to mitigate application performance in a manner to reduce heat generated by the first subsystem.

Application performance mitigation for wearable device

One disclosed example provides a computing device comprising a processing device and a storage device storing instructions executable by the processing device to execute in a first local process an application that outputs digital content for rendering and display. During execution of the application, the instructions are executable to provide, to a second local or remote process, object information regarding an object to be rendered by the second local or remote process, receive, from the second local or remote process, a rendering of the object, output the rendering of the object to display the object, receive a manipulation made to the object, provide, to the second local or remote process, updated object information based on the manipulation made to the object, receive, from the second local or remote process, an updated rendering of the object, and output the updated rendering of the object to display the object.

Hybrid rendering

A method for efficiently populating a display is provided. The method can include identifying a point at which a world ray intersects a capture surface defined by capture points of a scene, identifying a capture point closest to the identified point, generating a motion vector based on the motion vectors for each of two directly adjacent capture points, identifying a vector in the generated motion vector at a location at which the world ray intersects an image surface, and providing a pixel value from the image data of the capture point, the pixel value corresponding to a location in the image surface at which a vector of the generated motion vector points to the location at which the world ray intersects the image surface within a threshold distance or after a specified number of iterations.

Efficient image population from captured scene

Examples described herein generally relate to performing frame extrapolation in image frame rendering. A vertex mesh as a set of vertices is generated, and each vertex is mapped to a screen space position for defining a texture. One or more motion vectors for one or more regions in a first image frame of a stream of image frames can be determined. The screen space positions associated with at least a portion of the set of vertices within the texture can be modified based at least in part on the one or more motion vectors. A graphics processing unit (GPU) can render the first image frame into the texture. The extrapolated image frame is displayed after the first image frame and before a next image frame in the stream of image frames.

Techniques for extrapolating image frames

Encoding pixel information for pixels of an image. A method includes accessing information defining high-frequency image data correlated with pixels. The method further includes for each pixel, identifying if a vertex from the high-frequency image data is located in that pixel based on analysis of the high-frequency data correlated with the pixel. The method further includes, for one or more pixels in which a vertex is located, identifying the location of the vertex. The method further includes encoding the vertex location into image pixel data.

Vertex pixel buffer

In various embodiments, methods and systems reprojecting three-dimensional (3D) virtual scenes using discontinuity depth late stage reprojection are provided. A reconstruction point, that indicates camera pose information, is accessed. The reconstruction point is associated with a plurality of sample points of a three-dimensional (3D) virtual scene. One or more closest sample points, relative to the reconstruction point, are identified, from the plurality of sample points. Each of the one or more closest sample points is associated with a cube map of color data and depth data. A relative convergence score is determined for each of the one or more closest sample points based on performing a depth-aware cube map late stage reprojection operation in relation to the reconstruction point. A subset of the one or more closest sample points is identified based on the relative convergence score. A reconstructed 3D virtual image is generated using the subset.

Discontinuity-aware reprojection

A two-dimensional augmentation image is rendered from a three-dimensional model from a first virtual perspective. A transformation is applied to the augmentation image to yield an updated two-dimensional augmentation image that approximates a second virtual perspective of the three-dimensional model without additional rendering from the three-dimensional model.

Augmented-reality imaging

Examples are disclosed that relate to depth-aware late-stage reprojection. One example provides a computing system configured to receive and store image data, receive a depth map for the image data, processing the depth map to obtain a blurred depth map, and based upon motion data, determine a translation to be made to the image data. Further, for each pixel, the computing system is configured to translate an original ray extending from an original virtual camera location to an original frame buffer location to a reprojected ray extending from a translated camera location to a reprojected frame buffer location, determine a location at which the reprojected ray intersects the blurred depth map, and sample a color of a pixel for display based upon a color corresponding to the location at which the reprojected ray intersects the blurred depth map.

Depth-aware reprojection

In various embodiments, methods and systems for reprojecting images based on a velocity depth late stage reprojection process are provided. A reprojection engine supports reprojecting images based on an optimized late stage reprojection process that is performed based on both depth data and velocity data. Image data and corresponding depth and velocity data of the image data is received. A determination of an adjustment to be made to the image is made. The determination is made based on motion data, the depth data and the velocity data. The motion data corresponds to a device associated with displaying the image data. The velocity data supports determining calculated correction distances for portions of the image data. The image data is adjusted based on the determined adjustment. Adjusting the image data is based on integrating depth-data-based translation and velocity-data-based motion correction, into a single pass implementation, to adjust the image data.

Velocity and depth aware reprojection

Embodiments are described herein for determining a stabilization plane to reduce errors that occur when a homographic transformation is applied to a scene including 3D geometry and/or multiple non-coplanar planes. Such embodiments can be used, e.g., when displaying an image on a head mounted display (HMD) device, but are not limited thereto. In an embodiment, a rendered image is generated, a gaze location of a user is determined, and a stabilization plane, associated with a homographic transformation, is determined based on the determined gaze location. This can involve determining, based on the user's gaze location, variables of the homographic transformation that define the stabilization plane. The homographic transformation is applied to the rendered image to thereby generate an updated image, and at least a portion of the updated image is then displayed.

Stabilization plane determination based on gaze location

ClearType resolution recovery resampling of source content is performed as the source content is transformed from a source presentation format to a destination presentation format by at least distinguishing between pixels of a bitmap corresponding to the source content that are ClearType pixels and pixels of the same bitmap that are non-ClearType pixels. Identification of ClearType pixels is performed by identifying Alpha values of pixels that are less than 1, by identifying high frequency color fringing of pixels and/or by explicit declarations in the bitmap. The bitmap is resampled by applying transforms, such as convolution filters, in selective and discriminating ways, such that, for example, the ClearType pixels of the bitmap are resampled/transformed on a subpixel granularity and the non-ClearType pixels of the same bitmap are transformed on a pixel granularity.

Cleartype resolution recovery resampling

Methods for generating and displaying images associated with one or more virtual objects within an augmented reality environment at a frame rate that is greater than a rendering frame rate are described. The rendering frame rate may correspond with the minimum time to render images associated with a pose of a head-mounted display device (HMD). In some embodiments, the HMD may determine a predicted pose associated with a future position and orientation of the HMD, generate a pre-rendered image based on the predicted pose, determine an updated pose associated with the HMD subsequent to generating the pre-rendered image, generate an updated image based on the updated pose and the pre-rendered image, and display the updated image on the HMD. The updated image may be generated via a homographic transformation and/or a pixel offset adjustment of the pre-rendered image by circuitry within the display.

Reprojection OLED display for augmented reality experiences

One embodiment provides a method to display video such as computer-rendered animation or other video. The method includes assembling a sequence of video frames featuring a moving object, each video frame including a plurality of subframes sequenced for display according to a schedule. The method also includes determining a vector-valued differential velocity of the moving object relative to a head of an observer of the video. At a time scheduled for display of a first subframe of a given frame, first-subframe image content transformed by a first transform is displayed. At a time scheduled for display of the second subframe of the given frame, second-subframe image content transformed by a second transform is displayed. The first and second transforms are computed based on the vector-valued differential velocity to mitigate artifacts.

Avoidance of color breakup in late-stage re-projection

Growing community of inventors

Redmond, WA, United States of America

Ashraf A Michail