Microsoft Technology Licensing, LLC

Lijuan Qin

Xuedong D Huang

Nanshan Zeng

Andreas Stolcke

Zhuo Chen

Takuya Yoshioka

Dimitrios Basile Dimitriadis

William Isaac Hinthorn

Wei Xiong

Eyal Krupka

Amit Kumar Agarwal

Adi Diamant

Gregory Alan Howard

Susan Chory

Xiong Xiao

Shixiong Zhang

Karen Master Ben-Dor

Peter M Bergler

Ido Leichter

Tony He

Jon Arnett

Raz Halaly

Yoni Smolin

Aviv Hurvitz

Lingfeng Wu

Moshe David

Ilya Gurvich

Janis Jungeun Lee

Issa Yousef Khoury

Steve Seixeiro

Gurpreet Virdi

Petteri Mikkola

Michael Burt Goulding

David Jeremy Guy Wood

Lucas Scotta

A computer implemented method processes audio streams recorded during a meeting by a plurality of distributed devices. Operations include performing speech recognition on each audio stream by a corresponding speech recognition system to generate utterance-level posterior probabilities as hypotheses for each audio stream, aligning the hypotheses and formatting them as word confusion networks with associated word-level posteriors probabilities, performing speaker recognition on each audio stream by a speaker identification algorithm that generates a stream of speaker-attributed word hypotheses, formatting speaker hypotheses with associated speaker label posterior probabilities and speaker-attributed hypotheses for each audio stream as a speaker confusion network, aligning the word and speaker confusion networks from all audio streams to each other to merge the posterior probabilities and align word and speaker labels, and creating a best speaker-attributed word transcript by selecting the sequence of word and speaker labels with the highest posterior probabilities.

Speaker attributed transcript generation

A computer implemented method includes receiving audio signals representative of speech via multiple audio streams transmitted from corresponding multiple distributed devices, performing, via a neural network model, continuous speech separation for one or more of the received audio signals having overlapped speech, and providing the separated speech on a fixed number of separate output audio channels.

Processing overlapping speech from distributed devices

A computer implemented method includes receiving information streams on a meeting server from a set of multiple distributed devices included in a meeting, receiving audio signals representative of speech by at least two users in at least two of the information streams, receiving at least one video signal of at least one user in the information streams, associating a specific user with speech in the received audio signals as a function of the received audio and video signals, and generating a transcript of the meeting with an indication of the specific user associated with the speech.

Audio-visual diarization to identify meeting attendees

A method for facilitating a remote conference includes receiving a digital video and a computer-readable audio signal. A face recognition machine is operated to recognize a face of a first conference participant in the digital video, and a speech recognition machine is operated to translate the computer-readable audio signal into a first text. An attribution machine attributes the text to the first conference participant. A second computer-readable audio signal is processed similarly, to obtain a second text attributed to a second conference participant. A transcription machine automatically creates a transcript including the first text attributed to the first conference participant and the second text attributed to the second conference participant.

Computerized intelligent assistant for conferences

A computer implemented method includes receiving audio streams at a meeting server from two distributed devices that are streaming audio captured during an ad-hoc meeting between at least two users, comparing the received audio streams to determine that the received audio streams are representative of sound from the ad-hoc meeting, generating a meeting instance to process the audio streams in response to the comparing determining that the audio streams are representative of sound from the ad-hoc meeting, and processing the received audio streams to generate a transcript of the ad-hoc meeting.

Distributed device meeting initiation

Systems and methods for providing customized output based on a user preference in a distributed system are provided. In example embodiments, a meeting server or system receives audio streams from a plurality of distributed devices involved in an intelligent meeting. The meeting system identifies a user corresponding to a distributed device of the plurality of distributed devices and determines a preferred language of the user. A transcript from the received audio streams is generated. The meeting system translates the transcript into the preferred language of the user to form a translated transcript. The translated transcript is provided to the distributed device of the user.

Customized output to optimize for user preference in a distributed system

A computer implemented method includes receiving multiple channels of audio from three or more microphones detecting speech from a meeting of multiple users, localizing speech sources to determine an approximate direction of arrival of speech from a user, using a speech unmixing model to select two channels corresponding to a primary and a secondary microphone, and sending the two selected channels to a meeting server for generation of a speaker attributed meeting transcript.

Audio stream processing for distributed device meeting

A computer implemented method includes receiving audio signals representative of speech via multiple audio channels transmitted from corresponding multiple distributed devices, designating one of the audio channels as a reference channel, and for each of the remaining audio channels, determine a difference in time from the reference channel, and correcting each remaining audio channel by compensating for the corresponding difference in time from the reference channel.

Synchronization of audio signals from distributed devices

A digital assistant supported on computing devices such as smartphones, tablet computers, personal computers (PCs), wearable devices, game consoles, smart home devices and the like is configured with a notification system that interfaces with applications and/or other systems and resources so that the digital assistant may collect notifications across a range of a user's devices. The digital assistant may utilize a natural language interface (which may cooperate with touch and gesture interfaces) and applies awareness of context and user's behaviors to intelligently present the collected notifications on a single device for the user as a native digital assistant user experience. The digital assistant can sort notifications, group notifications together, and interact with the user to answer questions, provide information and status, and enable the user to action on a notification from the collection.

Interaction with notifications across devices with a digital assistant

Various user interfaces and other technologies for interacting with devices that support multiple communication lines can be implemented. Scenarios providing separate communications lines, such as voice over internet protocol (VOIP), social network communications, and the like can be supported. For example, communication-line-separated and communication-line-aggregated user interface paradigms can be supported. Intelligent selection of an appropriate paradigm can support user preferences, conversation user interfaces, and the like. Other features such as communication line defaults can help users deal with multiple communication line scenarios. A consistent, compact user interface for switching communication lines can be supported. Users can interact with their devices more efficiently and with less frustration. A wide variety of use scenarios are supported.

Intelligent user interfaces for multiple communication lines

Various user interfaces and other technologies for interacting with devices that support multiple SIM cards can be implemented. For example, SIM-separated and SIM-aggregated user interface paradigms can be supported. Intelligent selection of an appropriate paradigm can support user preferences, conversation user interfaces, and the like. Other features such as SIM card defaults can help users deal with multiple SIM card scenarios. A consistent, compact user interface for switching SIM cards can be supported. Users can interact with their devices more efficiently and with less frustration. A wide variety of use scenarios are supported.

Intelligent user interfaces for multiple SIM cards

Techniques and solutions are provided for adaptive input language switching. Automatic switching of a current input language can be performed by receiving an indication of a first language of displayed content (where the current input language is set to a second language different from the first language), receiving a selection of a text input area, and based on the selection, automatically switching the current input language from the second language to the first language. Switching of the current input language can also be performed by providing convenient user interface elements for selection by a user to automatically switch the current input language.

Adaptive input language switching

Growing community of inventors

Redmond, WA, United States of America

Lijuan Qin