Microsoft Technology Licensing, LLC

Hosam Adel Khalil

Tian Wang

Kazuhito Koishida

Wei-ge Chen

Xiaoqin Sun

Yifan Gong

Emilian Y Stoimenov

Christopher Hakan Basoglu

Khuram Shahid

Kshitiz Kumar

Guoli Ye

Amit Kumar Agarwal

Vladimir Cuperman

Guo-Wei Shieh

Sayan Dev Pathak

Chaojun Liu

Rui Zhao

Allen Gersho

Jian Wu

Kaustubh Prakash Kalgaonkar

Hong Wang

Veljko Miljanic

Warren Michael Lam

Ivaylo Andreanov Enchev

Anthony Phillip Stark

Mahadevan Srinivasan

Naveen Parihar

Hong Wang Sodoma

Wei-Ge Chen

Shuangyu Chang

Mu Han

Sang Kun Choe

Ajoy Nandi

Vinod Prakash

Ziad Al Bawab

Chun Hin Nelson Siu

Zoltan Romocsa

Carmen Quan

Anand U Desai

Huaming Wang

Feng Zhuge

Aadyot Bhatnagar

Denis Fisenko

Min Hu

Ziad Al-Bawab

Md Nizam Uddin Chy

Teng Yi

Sayan Pathak

Techniques are provided for early processing of a part of a user input to produce a response to the entire or final user input. While the user input is being received, a partial user input, which is a part of the final user input, is processed to produce a response. The response is a candidate response for the final user input. After the final user input is received, and if the partial user input is determined to match or be equivalent to the final user input, the first response, which is already available, is provided to one or more output devices for presentation. If the final user input is determined to differ from the partial user input, the final user input is processed to produce a second response to the final user input, and the second response is provided for presentation. In some instances, multiple partial user inputs are received and processed.

Processing part of a user input to produce an early response

Disclosed speech recognition techniques improve user-perceived latency while maintaining accuracy by: receiving an audio stream, in parallel, by a primary (e.g., accurate) speech recognition engine (SRE) and a secondary (e.g., fast) SRE; generating, with the primary SRE, a primary result; generating, with the secondary SRE, a secondary result; appending the secondary result to a word list; and merging the primary result into the secondary result in the word list. Combining output from the primary and secondary SREs into a single decoder as described herein improves user-perceived latency while maintaining or improving accuracy, among other advantages.

User-perceived latency while maintaining accuracy

Generally discussed herein are devices, systems, and methods for on-device detection of a wake word. A device can include a memory including model parameters that define a custom wake word detection model, the wake word detection model including a recurrent neural network transducer (RNNT) and a lookup table (LUT), the LUT indicating a hidden vector to be provided in response to a phoneme of a user-specified wake word, a microphone to capture audio, and processing circuitry to receive the audio from the microphone, determine, using the wake word detection model, whether the audio includes an utterance of the user-specified wake word, and wake up a personal assistant after determining the audio includes the utterance of the user-specified wake word.

On-device custom wake word detection

Generally discussed herein are devices, systems, and methods for custom wake word selection assistance. A method can include receiving, at a device, data indicating a custom wake word provided by a user, determining one or more characteristics of the custom wake word, determining that use of the custom wake word will cause more than a threshold rate of false detections based on the characteristics, rejecting the custom wake word as the wake word for accessing a personal assistant in response to determining that use of the custom wake word will cause more than a threshold rate of false detections, and setting the custom wake word as the wake word in response to determining that use of the custom wake word will not cause more than the threshold rate of false detections.

Wake word selection assistance architectures and methods

Acoustic features are batched into two different batches. The second batch of the two batches is made in response to a detection of a word hypothesis output by a speech recognition network that received the first batch. The number of acoustic feature frames of the second batch is equal to a second batch size greater than the first batch size. The second batch is also to the speech recognition network for processing.

Adaptive batching to reduce recognition latency

A system includes acquisition of a domain grammar, determination of an interpolated grammar based on the domain grammar and a base grammar, determination of a delta domain grammar based on an augmented first grammar and the interpolated grammar, determination of an out-of-vocabulary class based on the domain grammar and the base grammar, insertion of the out-of-vocabulary class into a composed transducer composed of the augmented first grammar and one or more other transducers to generate an updated composed transducer, composition of the delta domain grammar and the updated composed transducer, and application of the composition of the delta domain grammar and the updated composed transducer to an output of an acoustic model.

Online language model interpolation for automatic speech recognition

Embodiments may include collection of a first batch of acoustic feature frames of an audio signal, the number of acoustic feature frames of the first batch equal to a first batch size, input of the first batch to a speech recognition network, collection, in response to detection of a word hypothesis output by the speech recognition network, of a second batch of acoustic feature frames of the audio signal, the number of acoustic feature frames of the second batch equal to a second batch size greater than the first batch size, and input of the second batch to the speech recognition network.

Generally discussed herein are devices, systems, and methods for wake word verification. A method can include receiving, at a server, a message from a device indicating that an utterance of a user-defined wake word was detected at the device, the message including (a) audio samples or features extracted from the audio samples and (b) data indicating the user-defined wake word, retrieving or generating, at the server, a custom decoding graph for the user-defined wake word, wherein the decoding graph and the static portion of the wake word verification model form a custom wake word verification model for the user-defined wake word, executing the wake word verification model to determine a likelihood that the wake word was uttered, and providing a message to the device indicating whether wake was uttered based on the determined likelihood.

Online verification of custom wake word

The described technology provides arbitration between speech recognition results generated by different automatic speech recognition (ASR) engines, such as ASR engines trained according to different language or acoustic models. The system includes an arbitrator that selects between a first speech recognition result representing an acoustic utterance as transcribed by a first ASR engine and a second speech recognition result representing the acoustic utterance as transcribed by a second ASR engine. This selection is based on a set of confidence features that is initially used by the first ASR engine or the second ASR engine to generate the first and second speech recognition results.

Confidence features for automated speech recognition arbitration

A method of compensating for jitter in a packet stream is described. The method comprises placing undecoded frames extracted from packets in the packet stream into a jitter buffer while decoding frames from the jitter buffer and placing the decoded frames into a sample buffer at a rate determined using an average playout delay. The average playout delay is the running average of the playout delay calculated for each packet as each packet becomes available. The playout delay for each packet is the sum of a sample buffer delay and a jitter buffer delay. As each packet is received, the average playout delay is adjusted based on a comparison of the playout delay associated with the received packet to the current average playout delay.

Network jitter smoothing with reduced delay

A continuous comfort noise is provided that is overlaid for the entire duration of a conference call scenario. The comfort noise may be adapted to match the levels of the actual background noise detected on one or more of the conference call participant's devices on the transmitting end(s) of a conference call as well as the participants' speech levels. The comfort noise may also be adapted to the type of listening device employed on the receiving end of a conference call. The comfort noise level may be customized to an appropriate and comfortable level for the type of listening device being used, and the system may continuously mix the comfort noise with incoming audio signals for the entire duration of a conference call, lowering the comfort noise level gradually during speaking periods for additional user experience improvement.

Adaptive conference comfort noise

Growing community of inventors

Redmond, WA, United States of America

Hosam Adel Khalil