Microsoft Technology Licensing, LLC

Facebook Technologies, LLC

Meta Platforms, Inc.

Justin Avram Clark

Jeffrey Jesus Evertt

Christopher Harley Willoughby

Joel Deaguero

Relja Markovic

Mike Scavezze

Michael Jason Gourlay

David C Haley, Jr

Joe Sola

Daniel Joseph McCulloch

Arthur Charles Tomlin

Brian James Mount

Adam G Poulos

Anthony James Ambrus

Nicholas Colonnese

Jea Gon Park

Tristan Thomas Trutna

Sean Jason Keller

Kevin Geisner

Shawn Crispin Wright

Dan Osborn

Priyanshu Agarwal

David M Hill

Daniel Lee Osborn

Peter Glenn Sarrett

Mark Mihelich

Andrew R Campbell

Ben John Hindle

Michael A Spalding

Matthew J Puls

Zachary Tyler Middleton

Charles Everett Martin

Matthew Evan Pezent

Jessica Hartcher-O'Brien

Daniel J Mcculloch

Mark Thomas Mihelich

The disclosed method may include detecting motion of a user, estimating, for the detected motion of the user, effort expended by the user in performing the motion, determining, based on the detected motion and the estimation of expended effort, a haptic profile for conveying to the user a physical sensation of expending the effort, and simulating a sensation of expending the effort by executing the haptic profile in at least one haptic device that is worn by the user. Various other methods, systems, and/or computer-readable media are also disclosed.

Systems and methods for simulating a sensation of expending effort in a virtual environment

Technology is described for (3D) space carving of a user environment based on movement through the user environment of one or more users wearing a near-eye display (NED) system. One or more sensors of the NED system provide sensor data from which a distance and direction of movement can be determined. Spatial dimensions for a navigable path can be represented based on user height data and user width data of the one or more users who have traversed the path. Space carving data identifying carved out space can be stored in a 3D space carving model of the user environment. The navigable paths can also be related to position data in another kind of 3D mapping like a 3D surface reconstruction mesh model of the user environment generated from depth images.

Space carving based on human physical data

A method, medium, and virtual object for providing a virtual representation with an attribute are described. The virtual representation is generated based on a digitization of a real-world object. Properties of the virtual representation, such as colors, shape similarities, volume, surface area, and the like are identified and an amount or degree of exhibition of those properties by the virtual representation is determined. The properties are employed to identify attributes associated with the virtual representation, such as temperature, weight, or sharpness of an edge, among other attributes of the virtual object. A degree of exhibition of the attributes is also determined based on the properties and their degrees of exhibition. Thereby, the virtual representation is provided with one or more attributes that instruct presentation and interactions of the virtual representation in a virtual world.

Adding attributes to virtual representations of real-world objects

A tracking sensor integration system presented herein collects sensor data obtained for each time frame by a plurality of sensors attached to a wearable garment placed on a user's hand. A controller coupled to the tracking sensor integration system calculates a measurement gain based at least in part on collected sensor data, and determines prediction for a pose of the user's hand for the current time frame using the collected sensor data and a plurality of estimation parameters for the current time frame. The controller then updates the estimation parameters for the current time frame, based in part on the measurement gain and the prediction for the pose of the user's hand. The controller determines an estimated pose for the user's hand, based in part on the updated estimation parameters and the collected sensor data.

Tracking sensor integration system and method for recursive estimation of pose of user's body part

To digitize an object, a camera captures images of different sides of the object with color and depth data. At least two different sides of the object are identified from the images, and constructions are created of the sides of the object from the images. Points of the constructions to connect to one another are determined and used to align the constructions. The construction are merged to generate a rendition of the object. Various techniques are applied to extrapolate edges, remove seams, extend color intelligently, filter noise, apply skeletal structure to the object, and optimize the digitization further. The rendition of the object can be provided for display as a digital representation of the object and potentially used in different applications (e.g., games, Web, etc.).

Object digitization

Technology is described for (3D) space carving of a user environment based on movement through the user environment of one or more users wearing a near-eye display (NED) system. One or more sensors on the near-eye display (NED) system provide sensor data from which a distance and direction of movement can be determined. Spatial dimensions for a navigable path can be represented based on user height data and user width data of the one or more users who have traversed the path. Space carving data identifying carved out space can be stored in a 3D space carving model of the user environment. The navigable paths can also be related to position data in another kind of 3D mapping like a 3D surface reconstruction mesh model of the user environment generated from depth images.

Digitizing objects in a picture is discussed herein. A user presents the object to a camera, which captures the image comprising color and depth data for the front and back of the object. For both front and back images, the closest point to the camera is determined by analyzing the depth data. From the closest points, edges of the object are found by noting large differences in depth data. The depth data is also used to construct point cloud constructions of the front and back of the object. Various techniques are applied to extrapolate edges, remove seams, extend color intelligently, filter noise, apply skeletal structure to the object, and optimize the digitization further. Eventually, a digital representation is presented to the user and potentially used in different applications (e.g., games, Web, etc.).

Methods, systems, and computer-storage media having computer-usable instructions embodied thereon, for controlling objects in a virtual environment are provided. Real-world objects may be received into a virtual environment. The real-world objects may be any non-human object. An object skeleton may be identified and mapped to the object. A user skeleton of the real-world user may also be identified and mapped to the object skeleton. By mapping the user skeleton to the object skeleton, movements of the user control the movements of the object in the virtual environment.

Controlling objects in a virtual environment

Volumes of a 3D physical space are used in a surface reconstruction process, where adjacent volumes share vertices so that no gaps or overlaps between the volumes exist. As a result, a continuous surface is obtained in the surface reconstruction process. The vertices are anchored to nodes in a pose graph, such that locations of the vertices are adjusted as the pose graph is updated. As a result, a deformation of the volumes is permitted. Based on the deformation of a volume, a region of a depth map of the physical space is deformed correspondingly. Each vertex can be anchored to a closest node of the pose graph, or to a point which is based on a combination of nodes. In one approach, the point is defined based on the closest node and other nodes within a defined radius of the closest node.

Using free-form deformations in surface reconstruction

Depth maps of a physical space are obtained using a depth sensor carried by a rig such as a robot or a head mounted display device worn by a user. Visible light images are also obtained. The images and orientation readings are used to create a pose graph which includes nodes connected by links. The nodes are associated with different poses of the rig and the corresponding images. Links between the nodes represent correspondences between the images, and transforms between coordinate systems of the nodes. As new images are captured, the pose graph is updated to reduce an accumulation of errors. Furthermore, surfaces in the physical space can be reconstructed at any time according to the current state of the pose graph. Volumes used in a surface reconstruction process are anchored to the nodes such that the positions of the volumes are adjusted as the pose graph is updated.

Large-scale surface reconstruction that is robust against tracking and mapping errors

Digitizing objects in a picture is discussed herein. A user presents the object to a camera, which captures the image comprising color and depth data for the front and back of the object. The object is recognized and digitized using color and depth data of the image. The user's client queries a server managing images uploaded by other users for virtual renditions of the object, as recognized in the other images. The virtual renditions from the other images are merged with the digitized version of the object in the image captured by the user to create a composite rendition of the object.

Object refinement using many data sets

Systems, methods, and computer media for generating an avatar reflecting a player's current appearance. Data describing the player's current appearance is received. The data includes a visible spectrum image of the player, a depth image including both the player and a current background, and skeletal data for the player. The skeletal data indicates an outline of the player's skeleton. Based at least in part on the received data, one or more of the following are captured: a facial appearance of the player; a hair appearance of the player; a clothing appearance of the player; and a skin color of the player. A 3D avatar resembling the player is generated by combining the captured facial appearance, hair appearance, clothing appearance, and/or skin color with predetermined avatar features.

Generation of avatar reflecting player appearance

Growing community of inventors

Kirkland, WA, United States of America

Justin Avram Clark