Microsoft Technology Licensing, LLC

Honda Motor Co., Ltd.

Vivek Pradeep

Steven Nabil Bathiche

Karlton David Powell

Timothy Andrew Large

Stephen G Latta

Shahram Izadi

Kevin Geisner

Moshe Randall Lutz

Pablo Luis Sala

David Nister

Alice Jane Bernheim Brush

David Kim

Eric Chris Wolfgang Sommerlade

Vaibhav Thukral

Luis Eduardo Cabrera-Cordon

Ilya Dmitriyevich Zharkov

Liang Wang

Nathan James Luquetta-fish

Michael Bleyer

Kenneth Paul Hinckley

Sing Bing Kang

Hrvoje Benko

Jamie Daniel Joseph Shotton

Anthony Reed

Nigel Stuart Keam

Sudipta Narayan Sinha

Daniel G O'Neil

Neil Emerton

Elizabeth Picchietti Salowitz

Flavio Protasio Ribeiro

Ibrahim Eden

Xiaoyan Hu

Weerapan Wilairat

Jongwoo Lim

John Guido Atkins Weiss

David B Perry

Christoph Rhemann

Hamidreza Vaezi Joze

Alexander Norman Bennett

Christopher Zach

Sharath Viswanathan

John G A Weiss

Tsitsi Kolawole

John Weiss

Michelle Lynn Holtmann

Krzysztof Jan Luchowiec

Luming Liang

Carlos A C Pessoa

Matheen Siddiqui

Faezeh Amjadi

Mehran Khodabandeh

Quintus Travis

Jose Antonio Lara Silva

Luis E Cabrera-Cordon

Machine learning techniques are leveraged to provide personalized assistance on a computing device. In some configurations a timeline of a user's interactions with the computing device is generated. For example, screenshots and audio streams may be saved as entries in the timeline. Context—the state of the computing device when the entry is created, such as which documents and websites are open—is also stored. Entries in the timeline are processed by a model to generate embedding vectors. The timeline may be searched by finding the embedding vector that is closest to an embedding vector derived from a search query. The user may select a query result, causing the associated context to be restored. For example, if the query is 'show me all documents related to my upcoming trip to Japan', the query result may open documents and websites that were open when booking a flight to Japan.

User activity history experiences powered by a machine learning model

Methods and systems for generating and using a semantic index are provided. In some examples, content data is received. The content data includes a plurality of subsets of content data. Each of the plurality of subsets of content data are labelled, based on a semantic context corresponding to the content data. The plurality of subsets of content data and their corresponding labels are stored. The plurality of subsets of content data are grouped, based on their labels, thereby generating one or more groups of subsets of content data. Further, a computing device is adapted to perform an action, based on the one or more groups of subsets of content data.

Generating and using a semantic index

A computational photography system is described herein including a guidance system and a detail enhancement system. The guidance system uses a first neural network that maps an original image provided by an image sensor to a guidance image, which represents a color-corrected and lighting-corrected version of the original image. A combination unit combines the original image and the guidance image to produce a combined image. A detail-enhancement system then uses a second neural network to map the combined image to a predicted image. The predicted image supplements the guidance provided by the first neural network by sharpening details in the original image. A training system is also described herein for training the first and second neural networks. The training system alternates in the data it feeds the second neural network, first using a guidance image as input to the second neural network, and then using a corresponding ground-truth image.

Dual-stage system for computational photography, and technique for training same

Methods and systems for automatically generating training data for use in machine learning are disclosed. The methods can involve the use of environmental data derived from first and second environmental sensors for a single event. The environmental data types derived from each environmental sensor are different. The event is detected based on first environmental data derived from the first environmental sensor, and a portion of second environmental data derived from the second environmental sensor is selected to generate training data for the detected event. The resulting training data can be employed to train machine learning models.

Automated collection of machine learning training data

An optical system, comprising a multi-spectral optical element, a switchable filter, a dual bandpass filter, and a sensor. The multi-spectral optical element receives light in at least a first spectral band and a second spectral band. The dual bandpass filter filters out wavelengths of light in a transition region of the switchable filter between the first spectral band and the second spectral band. The switchable filter filters light received from the dual bandpass filter in the first spectral band in a first mode where the switchable filter transmits light in the first spectral band and in a second mode where the switchable filter does not transmit light in the first spectral band. The sensor is disposed at an image plane, and the multi-spectral optical element is configured to produce a modulation transfer function value that is a above a predetermined threshold for each of the spectral bands.

Switchable multi-spectrum optical sensor

An intelligent assistant device is configured to communicate non-verbal cues. Image data indicating presence of a human is received from one or more cameras of the device. In response, one or more components of the device are actuated to non-verbally communicate the presence of the human. Data indicating context information of the human is received from one or more of the sensors. Using at least this data one or more contexts of the human are determined, and one or more components of the device are actuated to non-verbally communicate the one or more contexts of the human.

Intelligent assistant device communicating non-verbal cues

Methods and apparatus for capturing motion from a self-tracking device are disclosed. In embodiments, a device self-tracks motion of the device relative to a first reference frame while recording motion of a subject relative to a second reference frame, the second reference frame being a reference frame relative to the device. In the embodiments, the subject may be a real object or, alternately, the subject may be a virtual subject and a motion of the virtual object may be recorded relative to the second reference frame by associating a position offset relative to the device with the position of the virtual object in the recorded motion. The motion of the subject relative to the first reference frame may be determined from the tracked motion of the device relative to the first frame and the recorded motion of the subject relative to the second reference frame.

Motion capture from a mobile self-tracking device

A first intelligent assistant computing device configured to receive and respond to natural language inputs provided by human users syncs to a reference clock of a wireless computer network. The first intelligent assistant computing device receives a communication sent by a second intelligent assistant computing device indicating a signal emission time at which the second intelligent assistant computing device emitted a position calibration signal. The first intelligent assistant computing device records a signal detection time at which the position calibration signal was detected. Based on a difference between 1) the signal emission time and the signal detection time, and 2) a known propagation speed of the position calibration signal, a distance between the first and second intelligent assistant computing devices is calculated.

Position calibration for intelligent assistant computing device

Computing devices and methods for associating a semantic identifier with an object are disclosed. In one example, a three-dimensional model of an environment comprising the object is generated. Image data of the environment is sent to a user computing device for display by the user computing device. User input comprising position data of the object and the semantic identifier is received. The position data is mapped to a three-dimensional location in the three-dimensional model at which the object is located. Based at least on mapping the position data to the three-dimensional location of the object, the semantic identifier is associated with the object.

Associating semantic identifiers with objects

Methods, apparatuses, and computer-readable mediums for generating human action data sets are disclosed by the present disclosure. In an aspect, an apparatus may receive a set of reference images, where each of the images within the set of reference images includes a person, and a background image. The apparatus may identify body parts of the person from the set of reference image and generate a transformed skeleton image by mapping each of the body parts of the person to corresponding skeleton parts of a target skeleton. The apparatus may generate a mask of the transformed skeleton image. The apparatus may generate, using machine learning, a frame of the person formed according to the target skeleton within the background image.

Human action data set generation in a machine learning system

Remote depth sensing techniques are described via relayed depth from diffusion. In one or more implementations, a remote depth sensing system is configured to sense depth as relayed from diffusion. The system includes an image capture system including an image sensor and an imaging lens configured to transmit light to the image sensor through an intermediate image plane that is disposed between the imaging lens and the image sensor, the intermediate plane having an optical diffuser disposed proximal thereto that is configured to diffuse the transmitted light. The system also includes a depth sensing module configured to receive one or more images from the image sensor and determine a distance to one or more objects in an object scene captured by the one or more images using a depth by diffusion technique that is based at least in part on an amount of blurring exhibited by respective said objects in the one or more images.

Remote depth sensing via relayed depth from diffusion

An entity-tracking computing system receives sensor information from a plurality of different sensors. The positions of entities detected by the various sensors are resolved to an environment-relative coordinate system so that entities identified by one sensor can be tracked across the fields of detection of other sensors.

Entity-tracking computing system

Resolution enhancement techniques are described. An apparatus may receive first image data at a first resolution, and second image data at a resolution less than the first resolution. The second image data may be scaled to the first resolution and compared to the first image data. Application of a neural network may scale the first image data to a resolution higher than the first resolution. The application of the neural network may incorporate signals based on the scaled second image data. The signals may include information obtained by comparing the scaled second image data to the resolution of the first image data.

Image resolution enhancement

A 3D silhouette sensing system is described which comprises a stereo camera and a light source. In an embodiment, a 3D sensing module triggers the capture of pairs of images by the stereo camera at the same time that the light source illuminates the scene. A series of pairs of images may be captured at a predefined frame rate. Each pair of images is then analyzed to track both a retroreflector in the scene, which can be moved relative to the stereo camera, and an object which is between the retroreflector and the stereo camera and therefore partially occludes the retroreflector. In processing the image pairs, silhouettes are extracted for each of the retroreflector and the object and these are used to generate a 3D contour for each of the retroreflector and object.

3D silhouette sensing system

Example embodiments simultaneously acquire multiple different focus state images for a scene at a video rate. The focus state images are acquired from a static arrangement of static optical elements. The focus state images are suitable for and sufficient for determining the depth of an object in the scene using depth from defocus (DFD) processing. The depth of the object in the scene is determined from the focus state images using DFD processing. The static optical elements may be off-the-shelf components that are used without modification. The static elements may include image sensors aligned to a common optical path, a beam splitter in the common optical path, and telecentric lenses that correct light in multiple optical paths produced by the beam splitter. The multiple optical paths may differ by a defocus delta. Simultaneous acquisition of the multiple focus state images facilitates mitigating motion blur associated with conventional DFD processing.

Computer vision depth sensing at video rate using depth from defocus

Growing community of inventors

Snohomish, WA, United States of America

Vivek Pradeep