Microsoft Technology Licensing, LLC

Jeffrey Jesus Evertt

Joel Deaguero

Justin Avram Clark

Relja Markovic

Christopher Harley Willoughby

David M Hill

Richard Carl Roesler

Emiko Valiane Charbonneau

Andrew William Jean

Alan M Jones

Mike Scavezze

Kevin Geisner

David C Haley, Jr

Joe Sola

Stephen G Latta

Darren Bennett

Semih Energin

Karim Audrey Luccin

James Dack

Charles W Carlson

Jedd Chevrier

Oscar Omar Garza Santos

Matthew Jon Paul

Dylan Vance

David Galloway

Shawn Crispin Wright

Dan Osborn

David Fields

Daniel Lee Osborn

Nazeeh Amin Eldirghami

Peter Glenn Sarrett

Kevin W Barnes

Mark Mihelich

Andrew R Campbell

Michael A Spalding

Ben John Hindle

Charles Everett Martin

Zachary Tyler Middleton

Matthew J Puls

Mark Thomas Mihelich

An improved human-computer interface ('HCI') is disclosed herein for viewing a three-dimensional (“3D”) representation of a real-world environment from different, changing, and/or multiple perspectives. An AR device may capture, in real-time, a 3D representation of a scene using a surface reconstruction (“SR”) camera and a traditional Red Green & Blue (“RGB”) camera. The 3D representation may be transmitted to and viewed on a user's computing device, enabling the user to navigate the 3D representation. The user may view the 3D representation in a free-third-person mode, enabling the user to virtually walk or fly through the representation captured by the AR device. The user may also select a floor plan mode for a top-down or isomorphic perspective. Enabling a user to view a scene from different perspectives enhances understanding, speeds trouble-shooting, and fundamentally improves the capability of the computing device, the AR device, and the combination thereof.

Re-creation of virtual environment through a video call

An improved human-computer interface ('HCI') is disclosed herein for viewing a three-dimensional ('3D') representation of a real-world environment from different, changing, and/or multiple perspectives. An AR device may capture, in real-time, a 3D representation of a scene using a surface reconstruction (&#x201c;SR&#x201d;) camera and a traditional Red Green & Blue (&#x201c;RGB&#x201d;) camera. The 3D representation may be transmitted to and viewed on a user's computing device, enabling the user to navigate the 3D representation. The user may view the 3D representation in a free-third-person mode, enabling the user to virtually walk or fly through the representation captured by the AR device. The user may also select a floor plan mode for a top-down or isomorphic perspective. Enabling a user to view a scene from different perspectives enhances understanding, speeds trouble-shooting, and fundamentally improves the capability of the computing device, the AR device, and the combination thereof.

The techniques introduced here enable a display system, such as an HMD device, to generate and display to a user a holographic structure matching a real-world structure. In some embodiments vertices, edges and planes of the holographic schematics are generated via the use of a peripheral tool that is positioned by a user. In other embodiments, other user input indicates the bounds of the holographic schematic. In response to user action, a holographic schematic is made to appear including corresponding real-world size measurements. The corresponding measurements are used to develop a holographic structure that integrates with the holographic schematic.

Mixed reality measurement with peripheral tool

A mixed reality display system determines a shared coordinate system that is understood by a mixed reality application running on the mixed reality display system and an operating system of the mixed reality display system. The operating system can display a system user interface (UI) element in a mixed reality environment. The system UI element can be displayed at a location in a mixed reality environment. The location is specified by the mixed reality application according to the shared coordinate system. A size and orientation for displaying the system UI element may also be specified. Also, the location, size and orientation may be specified through application program interfaces (API) of the operating system. API calls may be made per frame to adjust the location, size or orientation per frame of the displayed mixed reality environment.

Mixed reality system user interface placement

A method, medium, and virtual object for providing a virtual representation with an attribute are described. The virtual representation is generated based on a digitization of a real-world object. Properties of the virtual representation, such as colors, shape similarities, volume, surface area, and the like are identified and an amount or degree of exhibition of those properties by the virtual representation is determined. The properties are employed to identify attributes associated with the virtual representation, such as temperature, weight, or sharpness of an edge, among other attributes of the virtual object. A degree of exhibition of the attributes is also determined based on the properties and their degrees of exhibition. Thereby, the virtual representation is provided with one or more attributes that instruct presentation and interactions of the virtual representation in a virtual world.

Adding attributes to virtual representations of real-world objects

Technologies described herein provide a mixed environment display of attached control elements. The techniques disclosed herein enable users of a first computing device to interact with a remote computing device configured to control an object, such as a light, appliance, or any other suitable object. Configurations disclosed herein enable the first computing device to cause one or more actions, such as a selection of the object or the display of a user interface, by capturing and analyzing input data defining the performance of one or more gestures, such as a user looking at the object controlled by the second computing device. Rendered graphical elements configured to enable the control of the object can be displayed with a real-world view of the object.

Mixed environment display of attached control elements

To digitize an object, a camera captures images of different sides of the object with color and depth data. At least two different sides of the object are identified from the images, and constructions are created of the sides of the object from the images. Points of the constructions to connect to one another are determined and used to align the constructions. The construction are merged to generate a rendition of the object. Various techniques are applied to extrapolate edges, remove seams, extend color intelligently, filter noise, apply skeletal structure to the object, and optimize the digitization further. The rendition of the object can be provided for display as a digital representation of the object and potentially used in different applications (e.g., games, Web, etc.).

Object digitization

Concepts and technologies are described herein for providing enhanced configuration and control of robots. Configurations disclosed herein augment a mobile computing device, such as a robot, with resources for understanding and navigation of an environment surrounding the computing device. The resources can include sensors of a separate computing device, which may be in the form of a head-mounted display. Data produced by the resources can be used to generate instructions for the mobile computing device. The sensors of the separate computing device can also detect a change in an environment or a conflict in the actions of the mobile computing device, and dynamically modify the generated instructions. By the use of the techniques disclosed herein, a simple, low-cost robot can understand and navigate through a complex environment and appropriately interact with obstacles and other objects.

Enhanced configuration and control of robots

Concepts and technologies are described herein for providing a mixed environment display of robotic actions. In some configurations, techniques disclosed herein can execute a set of instructions or run a simulation of the set of instructions to generate model data defining actions of a robot based on a set of instructions. Using the model data, one or more computing devices may generate graphical data comprising a graphical representation of the robot performing tasks or actions based on an execution of the instructions. The graphical data can be in form of an animation showing a robots actions, which may include movement of the robot and the robot's interaction with other objects. Graphical elements showing the status of the robot or graphical representations of the instructions may be included in the graphical data. The graphical data may be displayed on an interface or communicated to one or more computers for further processing.

Mixed environment display of robotic actions

Digitizing objects in a picture is discussed herein. A user presents the object to a camera, which captures the image comprising color and depth data for the front and back of the object. For both front and back images, the closest point to the camera is determined by analyzing the depth data. From the closest points, edges of the object are found by noting large differences in depth data. The depth data is also used to construct point cloud constructions of the front and back of the object. Various techniques are applied to extrapolate edges, remove seams, extend color intelligently, filter noise, apply skeletal structure to the object, and optimize the digitization further. Eventually, a digital representation is presented to the user and potentially used in different applications (e.g., games, Web, etc.).

Methods, systems, and computer-storage media having computer-usable instructions embodied thereon, for controlling objects in a virtual environment are provided. Real-world objects may be received into a virtual environment. The real-world objects may be any non-human object. An object skeleton may be identified and mapped to the object. A user skeleton of the real-world user may also be identified and mapped to the object skeleton. By mapping the user skeleton to the object skeleton, movements of the user control the movements of the object in the virtual environment.

Controlling objects in a virtual environment

Digitizing objects in a picture is discussed herein. A user presents the object to a camera, which captures the image comprising color and depth data for the front and back of the object. The object is recognized and digitized using color and depth data of the image. The user's client queries a server managing images uploaded by other users for virtual renditions of the object, as recognized in the other images. The virtual renditions from the other images are merged with the digitized version of the object in the image captured by the user to create a composite rendition of the object.

Object refinement using many data sets

Growing community of inventors

Kirkland, WA, United States of America

Jeffrey Jesus Evertt