Microsoft Technology Licensing, LLC

Capital One Services, LLC

Xiaofei Wang

Takuya Yoshioka

Zhong Meng

Yashesh Gaur

Naoyuki Kanda

Xuankai Chang

Zhuo Chen

Kai-Wen Chen

Brian Donohue

Bethany Sehon

Albert T Zellers

Weidan Zhou

Xuemei Pan

Nirmal Kumar Raajan

Naresh Singh

Jinyu Li

Xiong Xiao

Min Tang

Huaming Wang

Sefik Emre Eskimez

Hemin Yang

Zirun Zhu

A hypothesis stitcher for speech recognition of long-form audio provides superior performance, such as higher accuracy and reduced computational cost. An example disclosed operation includes: segmenting the audio stream into a plurality of audio segments; identifying a plurality of speakers within each of the plurality of audio segments; performing automatic speech recognition (ASR) on each of the plurality of audio segments to generate a plurality of short-segment hypotheses; merging at least a portion of the short-segment hypotheses into a first merged hypothesis set; inserting stitching symbols into the first merged hypothesis set, the stitching symbols including a window change (WC) symbol; and consolidating, with a network-based hypothesis stitcher, the first merged hypothesis set into a first consolidated hypothesis. Multiple variations are disclosed, including alignment-based stitchers and serialized stitchers, which may operate as speaker-specific stitchers or multi-speaker stitchers, and may further support multiple options for differing hypothesis configurations.

Hypothesis stitcher for speech recognition of long-form audio

Aspects described herein may relate to methods, systems, and apparatuses that determine one or more categories associated with a dataset, or a portion thereof. The determination may be performed based on one or more tags associated with the dataset and/or a description associated with the dataset. Further, the determination may be performed by searching an ontology based on the one or more tags and/or the description. The determination may be performed by using a machine-learning model based on the one or more tags and/or the description. Once the one or more categories associated with the dataset are determined, the one or more categories may be used as a basis for modifying the dataset and/or validating the dataset.

Determining data categorizations based on an ontology and a machine-learning model

The disclosure herein describes using a transcript generation model for generating a transcript from a multi-speaker audio stream. Audio data including overlapping speech of a plurality of speakers is obtained and a set of frame embeddings are generated from audio data frames of the obtained audio data using an audio data encoder. A set of words and channel change (CC) symbols are generated from the set of frame embeddings using a transcript generation model. The CC symbols are included between pairs of adjacent words that are spoken by different people at the same time. The set of words and CC symbols are transformed into a plurality of transcript lines, wherein words of the set of words are sorted into transcript lines based on the CC symbols, and a multi-speaker transcript is generated based on the plurality of transcript lines. The inclusion of CC symbols by the model enables efficient, accurate multi-speaker transcription.

Training and using a transcript generation model on a multi-speaker audio stream

Systems and methods are provided for generating and operating a speech enhancement model optimized for generating noise-suppressed speech outputs for improved human listening and live captioning. A computing system obtains a speech enhancement model trained on a first training dataset to generate noise-suppressed speech outputs and an automatic speech recognition model trained on a second training dataset to generate transcription labels for spoken language utterances. A third training dataset comprising a set of spoken language utterances is applied to the speech enhancement model to obtain a first noise-suppressed speech output which is applied to the automatic speech recognition model to generate a noise-suppressed transcription output for the set of spoken language utterances. Speech enhancement model parameters are updated to optimize the speech enhancement model to generate optimized noise-suppressed speech outputs based on a comparison of the noise-suppressed transcription output and ground truth transcription labels.

Systems and methods for human listening and live captioning

Growing community of inventors

Bellevue, WA, United States of America

Xiaofei Wang