International Business Machines Corporation

The Ohio State University

Eric Fosler-Lussier

Samuel Jonathan Thomas

Hong-Kwang Jeff Kuo

Brian E D Kingsbury

Vishal Sunder

George Andrei Saon

Jatin Ganhotra

Systems, computer-implemented methods, and computer program products to facilitate fine-grained textual knowledge transfer to improve speech recognition and understanding are provided. According to an embodiment, a system can comprise a processor that executes components stored in memory. The computer executable components comprise deriving component that can derive one or more speech-based embeddings from an utterance via a speech encoder. The computer executable components can comprise a cross-attention component that can align, at a token level, one or more large language model (LLM) based sentence embeddings with the one or more speech-based embeddings. The computer executable components can comprise a loss component that can combine an alignment loss and an automatic speech recognition (ASR) loss.

Textual knowledge transfer for improved speech recognition and understanding

Systems, computer-implemented methods, and computer program products to facilitate end to end integration of dialogue history for spoken language understanding are provided. According to an embodiment, a system can comprise a processor that executes components stored in memory. The computer executable components comprise a conversation component that encodes speech-based content of an utterance and text-based content of the utterance into a uniform representation.

End-to-end integration of dialog history for spoken language understanding

Growing community of inventors

Columbus, OH, United States of America

Eric Fosler-Lussier