Microsoft Technology Licensing, LLC

Harpreet Singh Sawhney

Ning Xu

Muhammad Zeeshan Zia

Amol Ashok Ambardekar

Pedro Urbina Escos

Bugra Tekin

Michael Ebstyne

Emanuel Shalev

Sheng Kai Tang

Andrey Konin

William Douglas Guyman

Jonathan C Hanzelka

Federica Bogo

Marc André Léon Pollefeys

Moses Obadeji Olafenwa

Bilha-Catherine W Githinji

Huseyin Coskun

Examples are disclosed that relate to computer-based tracking of a process performed by a user. In one example, multi-modal sensor information is received via a plurality of sensors. A world state of a real-world physical environment and a user state in the real-world physical environment are tracked based on the multi-modal sensor information. A process being performed by the user within a working domain is recognized based on the world state and the user state. A current step in the process is detected based on the world state and the user state. Domain-specific instructions directing the user how to perform an expected action are presented via a user interface device. A user action is detected based on the world state and the user state. Based on the user action differing from the expected action, domain-specific guidance to perform the expected action is presented via the user interface device.

Multi-modal sensor based process tracking and guidance

Examples are disclosed that relate to representing recorded hand motion. One example provides a computing device comprising a logic subsystem and a storage subsystem comprising instructions executable by the logic subsystem to receive a recorded representation of hand motion determined relative to a virtual model aligned to a first instance of an object, receive image data corresponding to an environment, and recognize a second instance of the object in the environment. The instructions are further executable to align the virtual model to the second instance of the object, and output a parametric representation of hand motion for display relative to the virtual model as aligned to the second instance of the object, such that the parametric representation is spatially consistent with the recorded representation of hand motion relative to the virtual model as aligned to the first instance of the object.

Spatially consistent representation of hand motion

An image data annotation system automatically annotates a physical object within individual images frames of an image sequence with relevant object annotations based on a three-dimensional (3D) model of the physical object. Annotating the individual image frames with object annotations includes updating individual image frames within image input data to generate annotated image data that is suitable for reliably training a DNN object detection architecture. Exemplary object annotations that the image data annotation system can automatically apply to individual image frames include, inter alia, object pose, image pose, object masks, 3D bounding boxes composited over the physical object, 2D bounding boxes composited over the physical object, and/or depth map information. Annotating the individual image frames may be accomplished by aligning the 3D model of the physical object with a multi-view reconstruction of the physical object that is generated by inputting an image sequence into a Structure-from-Motion and/or Multi-view Stereo pipeline.

Aligning input image data with model input data to generate image annotations

A method for object recognition includes, at a computing device, receiving an image of a real-world object. An identity of the real-world object is recognized using an object recognition model trained on a plurality of computer-generated training images. A digital augmentation model corresponding to the real-world object is retrieved, the digital augmentation model including a set of augmentation-specific instructions. A pose of the digital augmentation model is aligned with a pose of the real-world object. An augmentation is provided, the augmentation associated with the real-world object and specified by the augmentation-specific instructions.

Real-world object recognition for computing device

A computing device, including a processor configured to receive a first video including a plurality of frames. For each frame, the processor may determine that a target region of the frame includes a target object. The processor may determine a surrounding region within which the target region is located. The surrounding region may be smaller than the frame. The processor may identify one or more features located in the surrounding region. From the one or more features, the processor may generate one or more manipulated object identifiers. For each of a plurality of pairs of frames, the processor may determine a respective manipulated object movement between a first manipulated object identifier of the first frame and a second manipulated object identifier of the second frame. The processor may classify at least one action performed in the first video based on the plurality of manipulated object movements.

Action classification based on manipulated object movement

The disclosure herein describes training a machine learning model to recognize a real-world object based on generated virtual scene variations associated with a model of the real-world object. A digitized three-dimensional (3D) model representing the real-world object is obtained and a virtual scene is built around the 3D model. A plurality of virtual scene variations is generated by varying one or more characteristics. Each virtual scene variation is generated to include a label identifying the 3D model in the virtual scene variation. A machine learning model may be trained based on the plurality of virtual scene variations. The use of generated digital assets to train the machine learning model greatly decreases the time and cost requirements of creating training assets and provides training quality benefits based on the quantity and quality of variations that may be generated, as well as the completeness of information included in each generated digital asset.

Generating synthetic digital assets for a virtual scene including a model of a real-world object

Techniques for localizing a device based on images captured by the device. The techniques include receiving, from a device via a data communication network, image frame data for frames captured by an imaging camera included in the device, the frames including a first frame, automatically detecting real-world objects captured in the image frame data, automatically classifying the detected real-world objects as being associated with respective object classes, automatically identifying object class and instance dependent keypoints for the real-world objects based on the object classes associated with the real-world objects, and estimating a pose of the device for the first frame based on at least the identified object class and instance dependent keypoints.

Growing community of inventors

Kirkland, WA, United States of America

Harpreet Singh Sawhney