Microsoft Technology Licensing, LLC

Yue Dong

Xin Tong

Guojun Chen

Yingnan Ju

Pingchao Yu

Baining Guo

Zhouchen Lin

Lin Liang

Stephen S Lin

Jiaping Wang

Simon Stachniak

Jian Shi

Implementations of the subject matter described herein relate to mixed reality object rendering based on ambient light conditions. According to the embodiments of the subject matter described herein, while rendering an object a wearable computing device acquires light conditions of the real world, thereby increasing the reality of the rendered object. In particular, the wearable computing deice is configured to acquire an image of an environment where the wearable computing deice is located. The image is adjusted based on a cement parameter used when the image is captured. Subsequently, ambient light information is determined based on the adjusted image. In this way, the wearable computing deice can obtain more real and accurate emblem light information, so as to render to the user an object with enhanced reality. Accordingly, the user can have a better interaction experience.

Ambient light based mixed reality object rendering

Implementations of the subject matter described herein relate to mixed reality object rendering based on ambient light conditions. According to the embodiments of the subject matter described herein, while rendering an object, a wearable computing device acquires light conditions of the real world, thereby increasing the reality of the rendered object. In particular, the wearable computing device is configured to acquire an image of an environment where the wearable computing device is located. The image is adjusted based on a camera parameter used when the image is captured. Subsequently, ambient light information is determined based on the adjusted image. In this way, the wearable computing device can obtain more real and accurate ambient light information, so as to render to the user an object with enhanced reality. Accordingly, the user can have a better interaction experience.

Implementations of the subject matter described herein relate to mixed reality rendering of objects. According to the embodiments of the subject matter described herein, while rendering an object, a wearable computing device takes lighting conditions in the real world into account, thereby increasing the reality of the rendered object. In particular, the wearable computing device acquires environment lighting information of an object to be rendered and renders the object to a user based on the environment lighting information. In this way, the object rendered by the wearable computing device can be more real and accurate. The user will thus have a better interaction experience.

Mixed reality object rendering based on environment lighting

In accordance with implementations of the subject matter described herein, there is provided a solution of lighting estimation. In the solution, an input image about a real object and a depth map corresponding to the input image are obtained. A geometric structure of the scene in the input image is determined based on the depth map. Shading and shadow information on the real object caused by a light source in the scene is determined based on the determined geometric structure of the scene. Then, a lighting condition in the scene caused by the light source is determined based on the input image and the shading and shadow information. The virtual object rendered using the lighting condition obtained according to the solution can exhibit a realistic effect consistent with the real object.

Lighting estimation using an input image and depth map

Techniques and constructs can determine an albedo map and a shading map from a digital image. The albedo and shading maps can be determined based at least in part on a color-difference threshold. A color shading map can be determined based at least in part on the albedo map, and lighting coefficients determined based on the color shading map. The digital image can be adjusted based at least in part on the lighting coefficients. In some examples, respective shading maps can be produced for individual color channels of the digital image. The color shading map can be produced based at least in part on the shading maps. In some examples, a plurality of regions of the digital image can be determined, as can proximity relationships between individual regions. The albedo shading maps can be determined based at least in part on the proximity relationships.

Machine vision processing system

A 'Scene Re-Lighter' provides various techniques for using an automatically reconstructed light transport matrix derived from a sparse sampling of images to provide various combinations of complex light transport effects in images, including caustics, complex occlusions, inter-reflections, subsurface scattering, etc. More specifically, the Scene Re-Lighter reconstructs the light transport matrix from a relatively small number of acquired images using a 'Kernel Nystr&#xf6;m' based technique adapted for low rank matrices constructed from sparsely sampled images. A &#x201c;light transport kernel&#x201d; is incorporated into the Nystr&#xf6;m method to exploit nonlinear coherence in the light transport matrix. Further, an adaptive process is used to efficiently capture the sparsely sampled images from a scene. The Scene Re-Lighter is capable of achieving good reconstruction of the light transport matrix with only few hundred images to produce high quality relighting results. Further, the Scene Re-Lighter is also effective for modeling scenes with complex lighting effects and occlusions.

Growing community of inventors

Redmond, WA, United States of America

Yue Dong