Samsung Electronics Co., Ltd.

Apple Inc.

Qualcomm Incorporated

Yingen Xiong

Christopher A Peri

Lu Luo

Jared A Crawford

Marco Pontil

Ning Bi

Euisuk Chung

Lei Xu

Shuxue Quan

Tao Zhang

Yunke Pan

Haihui Guan

Divi Schmidt

Ravindraraj Mamadgi

A method includes obtaining images of a scene and corresponding position data of a device that captures the images. The method also includes determining position data and direction data associated with camera rays passing through keyframes of the images. The method further includes using a position-dependent multilayer perceptron (MLP) and a direction-dependent MLP to create sparse feature vectors. The method also includes storing the sparse feature vectors in at least one data structure. The method further includes receiving a request to render the scene on an augmented reality (AR) device associated with a viewing direction. In addition, the method includes rendering the scene associated with the viewing direction using the sparse feature vectors in the at least one data structure.

Real-time photorealistic view rendering on augmented reality (AR) device

A method includes obtaining multiple see-through image frames of an environment around an augmented reality (AR) device using multiple imaging sensors of the AR device. The method also includes generating a depth map based on the see-through image frames and generating a three-dimensional (3D) representation of the environment based on the depth map. The method further includes projecting the 3D representation onto a curved surface, mapping points of the projected 3D representation to multiple virtual view images, and presenting the virtual view images on one or more displays of the AR device. Generating the depth map may include generating an initial depth map using a trained machine learning model and modifying the initial depth map to provide both spatial consistency and temporal consistency in order to generate a refined depth map. The curved surface may include a portion of a cylindrical, spherical, or conical surface.

Generation and rendering of extended-view geometries in video see-through (VST) augmented reality (AR) systems

A method includes, in response to initiating a shooting mode of a camera application on an electronic device, collecting sensor information comprising at least one of: motion data of the electronic device, position data of the electronic device, and image data captured by one or more imaging sensors of the electronic device, wherein the shooting mode represents at least one of: a video record mode and an image capture mode. The method also includes determining, using a trained machine learning model, whether a user intention is to record video or capture an image based on features extracted from the sensor information. The method further includes recording video regardless of the shooting mode in response to determining that the user intention is to record video or capturing the image regardless of the shooting mode in response to determining that the user intention is to capture the image.

System and method for detecting a user intent to start a video recording

A method includes capturing an image and associating the image with a camera pose for each of multiple cameras. The method also includes determining, for each camera, a first contribution of the image for a first virtual view for display on a first display and a second contribution of the image for a second virtual view for display on a second display. The method further includes determining, for each camera, a first confidence map for the first virtual view based on the camera pose and a position of the camera in relation to a first virtual camera and a second confidence map for the second virtual view based on the camera pose and the position of the camera in relation to a second virtual camera. In addition, the method includes generating the first virtual view by combining the first contribution using the first confidence map for each of the cameras and the second virtual view by combining the second contribution using the second confidence map for each of the cameras.

Methods and devices for video rendering for video see-through (VST) augmented reality (AR)

A method includes receiving first and second images from first and second see-through cameras with first and second camera viewpoints. The method also includes generating a first virtual image corresponding to a first virtual viewpoint by applying a first mapping to the first image. The first mapping is based on relative positions of the first camera viewpoint and the first virtual viewpoint corresponding to a first eye of a user. The method further includes generating a second virtual image corresponding to a second virtual viewpoint by applying a second mapping to the second image. The second mapping is based on relative positions of the second camera viewpoint and the second virtual viewpoint corresponding to a second eye of the user. In addition, the method includes presenting the first and second virtual images to the first and second virtual viewpoints on at least one display panel of an augmented reality device.

Method and device for direct passthrough in video see-through (VST) augmented reality (AR)

A method includes obtaining images of an environment captured by imaging sensors associated with a passthrough AR device and position data and depth data associated with the images. The method also includes generating a point cloud representative of the environment based on the images, position data, and depth data. The method further includes generating a mesh for a specified image. The mesh includes grid points at intersections of mesh lines. The method also includes determining one or more depths of one or more grid points of the mesh. The method further includes transforming the mesh from a viewpoint of a specified imaging sensor that captured the specified image to a user viewpoint of the passthrough AR device based on the depth(s) of the grid point(s). In addition, the method includes rendering a virtual view of the specified image for presentation by the passthrough AR device based on the transformed mesh.

Mesh transformation with efficient depth reconstruction and filtering in passthrough augmented reality (AR) systems

A method includes obtaining first and second image data of a real-world scene, performing feature extraction to obtain first and second feature maps, and performing pose tracking based on at least one of the first image data, second image data, and pose data to obtain a 6DOF pose of an apparatus. The method also includes generating, based on the 6DOF pose, first feature map, and second feature map, a disparity map between the image data and generating an initial depth map based on the disparity map. The method further includes generating a dense depth map based on the initial depth map and a camera model and generating, based on the dense depth map, a three-dimensional reconstruction of at least pail of the scene. In addition, the method includes rendering an AR or XR display that includes one or more virtual objects positioned to contact one or more surfaces of the reconstruction.

System and method for depth and scene reconstruction for augmented reality or extended reality devices

In one embodiment, a method includes identifying, in each image of a stereoscopic pair of images of a scene at a particular time, every pixel as either a static pixel corresponding to a portion of a scene that does not have local motion at that time or a dynamic pixel corresponding to a portion of a scene that has local motion at that time. For each static pixel, the method includes comparing each of a plurality of depth calculations for the pixel, and when the depth calculations differ by at least a threshold amount, then re-labeling that pixel as a dynamic pixel. For each dynamic pixel, the method includes comparing a geometric 3D calculation for the pixel with a temporal 3D calculation for that pixel, and when the geometric 3D calculation and the temporal 3D calculation are within a threshold amount, then re-labeling the pixel as a static pixel.

Reconstructing a three-dimensional scene

A system and method for adaptive volume-based scene reconstruction for XR platform application are provided. The system includes an image sensor and a processor to perform the method for display distortion calibration. The method includes determining a processor computation load. The method also includes, based on the determined computation load, adjusting one or more parameters for the 3D scene reconstruction to compensate for the determined computation load. The method further includes rendering a reconstructed 3D scene.

System and method for adaptive volume-based scene reconstruction for xr platform

A method includes obtaining, from an image sensor, image data of a real-world scene; obtaining, from a depth sensor, sparse depth data of the real-world scene; and passing the image data to a first neural network to obtain one or more object regions of interest (ROIs) and one or more feature map ROIs. Each object ROI includes at least one detected object. The method also includes passing the image data and sparse depth data to a second neural network to obtain one or more dense depth map ROIs; aligning the one or more object ROIs, one or more feature map ROIs, and one or more dense depth map ROIs; and passing the aligned ROIs to a fully convolutional network to obtain a segmentation of the real-world scene. The segmentation contains one or more pixelwise predictions of one or more objects in the real-world scene.

Method and apparatus for scene segmentation for three-dimensional scene reconstruction

A method includes obtaining, from a memory of an electronic device connected to a head mounted display (HMD), a first reference frame, wherein the first reference frame comprises a first set of pixels associated with a first time. The method includes, rendering, at the electronic device, a source image as a new frame, wherein the new frame includes a second set of pixels associated with a display to be provided by the HMD at a second time, and generating, by the electronic device, a differential frame, wherein the differential frame is based on a difference operation between pixels of the new frame with pixels of the first reference frame to identify pixels unique to the new frame. Still further, the method includes sending the differential frame to the HMD, and storing the new frame in the memory of the electronic device as a second reference frame.

System and method for reducing image data throughput to head mounted display

A method includes receiving, from a camera, one or more frames of image data of a scene comprising a background and one or more three-dimensional objects, wherein each frame comprises a raster of pixels of image data; detecting layer information of the scene, wherein the layer information is associated with a depth-based distribution of the pixels in the one or more frames; and determining a multi-layer model for the scene, the multi-layer model comprising a plurality of discrete layers comprising first and second discrete layers, wherein each discrete layer is associated with a unique depth value relative to the camera. The method further includes mapping the pixels to the layers of the plurality of discrete layers; rendering the pixels as a first image of the scene as viewed from a first perspective; and rendering the pixels as a second image of the scene as viewed from a second perspective.

System and method for generating a three-dimensional photographic image

A method of video transformation for a video see-through (VST) augmented reality (AR) device includes obtaining video frames from multiple cameras associated with the VST AR device, where each video frame is associated with position data. The method also includes generating camera viewpoint depth maps associated with the video frames based on the video frames and the position data. The method further includes performing depth re-projection to transform the video frames from camera viewpoints to rendering viewpoints using the camera viewpoint depth maps. The method also includes performing hole filling of one or more holes created in one or more occlusion areas of at least one of the transformed video frames during the depth re-projection to generate at least one hole-filled video frame. In addition, the method includes displaying the transformed video frames including the at least one hole-filled video frame on multiple displays associated with the VST AR device.

Method and system for video transformation for video see-through augmented reality

A method includes receiving depth data of a real-world scene from a depth sensor, receiving image data of the scene from an image sensor, receiving movement data of the depth and image sensors from an IMU, and determining an initial 6DOF pose of an apparatus based on the depth data, image data, and/or movement data. The method also includes passing the 6DOF pose to a back end to obtain an optimized pose and generating, based on the optimized pose, image data, and depth data, a three-dimensional reconstruction of the scene. The reconstruction includes a dense depth map, a dense surface mesh, and/or one or more semantically segmented objects. The method further includes passing the reconstruction to a front end and rendering, at the front end, an XR frame. The XR frame includes a three-dimensional XR object projected on one or more surfaces of the scene.

System and method for three-dimensional scene reconstruction and understanding in extended reality (XR) applications

A system and method for 3D reconstruction with plane and surface reconstruction, scene parsing, depth reconstruction with depth fusion from different sources. The system includes display and a processor to perform the method for 3D reconstruction with plane and surface reconstruction. The method includes dividing a scene of an image frame into one or more plane regions and one or more surface regions. The method also includes generating reconstructed planes by performing plane reconstruction based on the one or more plane regions. The method also includes generating reconstructed surfaces by performing surface reconstruction based on the one or more surface regions. The method further includes creating the 3D scene reconstruction by integrating the reconstructed planes and the reconstructed surfaces.

Growing community of inventors

Mountain View, CA, United States of America

Yingen Xiong