Microsoft Technology Licensing, LLC

Ishani Chakraborty

Pedro Urbina Escos

Jonathan C Hanzelka

Thomas M Soemo

Lu Yuan

Gang Hua

Yi-Ling Chen

Gavin Jancke

Vibhav Vineet

Vicente A Rivera

A method for virtual object pose rendering includes receiving a digital image of a real-world object in a real-world environment. An imaged pose of the real-world object is estimated based at least in part on a neural radiance model encoding 3D (three-dimensional) spatial data for the real-world object, the neural radiance model derived from a plurality of previously captured digital images of the real-world object. A view of a virtual object corresponding to the real-world object is rendered based at least in part on the neural radiance model, such that the virtual object has a virtual pose consistent with the imaged pose of the real-world object in the digital image.

Virtual pose rendering using neural radiance model

Training a multi-object tracking model includes: generating a plurality of training images based at least on scene generation information, each training image comprising a plurality of objects to be tracked; generating, for each training image, original simulated data based at least on the scene generation information, the original simulated data comprising tag data for a first object; locating, within the original simulated data, tag data for the first object, based on at least an anomaly alert (e.g., occlusion alert, proximity alert, motion alert) associated with the first object in the first training image; based at least on locating the tag data for the first object, modifying at least a portion of the tag data for the first object from the original simulated data, thereby generating preprocessed training data from the original simulated data; and training a multi-object tracking model with the preprocessed training data to produce a trained multi-object tracker.

Training multi-object tracking models using simulation

The disclosure herein enables tracking of multiple objects in a real-time video stream. For each individual frame received from the video stream, a frame type of the frame is determined. Based on the individual frame being an object detection frame type, a set of object proposals is detected in the individual frame, associations between the set of object proposals and a set of object tracks are assigned, and statuses of the set of object tracks are updated based on the assigned associations. Based on the individual frame being an object tracking frame type, single-object tracking is performed on the frame based on each object track of the set of object tracks and the set of object tracks is updated based on the performed single-object tracking. For each frame received, a real-time object location data stream is provided based on the set of object tracks.

Tracking multiple objects in a video stream using occlusion-aware single-object tracking

A computing system is configured to train an object classifier. Monocular image data and ground-truth data are received for a scene. Geometric context is determined including a three-dimensional camera position relative to a fixed plane. Regions of interest (RoI) and a set of potential occluders are identified within the image data. For each potential occluder, an occlusion zone is projected onto the fixed plane in three-dimensions. A set of occluded RoIs on the fixed plane are generated for each occlusion zone. Each occluded RoI is projected back to the image data in two-dimensions. The classifier is trained by minimizing a loss function generated by inputting information regarding the RoIs and the occluded RoIs into the classifier, and by minimizing location errors of each RoI and each occluded RoI of the set on the fixed plane based on the ground-truth data. The trained classifier is then output for object detection.

Growing community of inventors

Seattle, WA, United States of America

Ishani Chakraborty