Google Inc.

Martin Baeuml

Gleb Skobeltsyn

Alexandru Ovidiu Dovlecel

Dmitry Osmakov

Vladimir Vuskovic

Stephan Wenger

Zineb Ait Bahajji

Evgeny A Cherepanov

Olga Kapralova

Thushan Amarasiriwardena

Behshad Behzadi

Roberto Pieraccini

Gianluca Martini

Quoc V Le

Noam M Shazeer

Trevor D Strohman

Vikram Sridar

Daniel De Freitas Adiwardana

Carl-Anton Ingmarsson

Alexander Bailey

Florent D'Halluin

Jonas Bragagnolo

Implementations relate to dialog management of a large language model (LLM) utilized in generating natural language (NL) output during an ongoing dialog. Processor(s) of a system can: receive NL based input as part of the ongoing dialog, generate NL based output utilizing the LLM, and cause the NL based output to be rendered. Further, the processor(s) can receive subsequent NL based input as part of the ongoing dialog. In some implementations, the processor(s) can determine whether to modify a corresponding dialog context in generating subsequent NL based output, and modify the corresponding dialog context accordingly. For example, the processor(s) can restrict the corresponding dialog context, or supplant the corresponding dialog context with a corresponding curated dialog context. In additional or alternative implementations, the processor(s) can modify a corresponding NL based output threshold utilized in generating the subsequent NL based response to ensure the resulting NL based output is desirable.

Dialog management for large language model- based (LLM-based) dialogs

Implementations relate to dynamically adapting a given assistant output based on a given persona, from among a plurality of disparate personas, assigned to an automated assistant. In some implementations, the given assistant output can be generated and subsequently adapted based on the given persona assigned to the automated assistant. In other implementations, the given assistant output can be generated specific to the given persona and without having to subsequently adapt the given assistant output to the given persona. Notably, the given assistant output can include a stream of textual content to be synthesized for audible presentation to the user, and a stream of visual cues utilized in controlling a display of a client device and/or in controlling a visualized representation of the automated assistant. Various implementations utilize large language models (LLMs), or output previously generated utilizing LLMs, to reflect the given persona in the given assistant output.

Dynamically adapting given assistant output based on a given persona assigned to an automated assistant

Methods, apparatus, and computer readable media are described related to automated assistants that proactively incorporate, into human-to-computer dialog sessions, unsolicited content of potential interest to a user. In various implementations, based on content of an existing human-to-computer dialog session between a user and an automated assistant, an entity mentioned by the user or automated assistant may be identified. Fact(s)s related to the entity or to another entity that is related to the entity may be identified based on entity data contained in database(s). For each of the fact(s), a corresponding measure of potential interest to the user may be determined. Unsolicited natural language content may then be generated that includes one or more of the facts selected based on the corresponding measure(s) of potential interest. The automated assistant may then incorporate the unsolicited content into the existing human-to-computer dialog session or a subsequent human-to-computer dialog session.

Proactive incorporation of unsolicited content into human-to-computer dialogs

As part of a dialog session between a user and an automated assistant, implementations can receive a stream of audio data that captures a spoken utterance including an assistant query, determine, based on processing the stream of audio data, a set of assistant outputs that are each predicted to be responsive to the assistant query, process, using large language model (LLM) output(s), the assistant outputs and context of the dialog session to generate a set of modified assistant outputs, and cause given modified assistant output, from among the set of modified assistant outputs, to be provided for presentation to the user in response to the spoken utterance. In some implementations, the LLM output(s) can be generated in an offline manner for subsequent use in an online manner. In additional or alternative implementations, the LLM output(s) can be generated in an online manner when the spoken utterance is received.

Using large language model(s) in generating automated assistant response(s

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for speech recognition. One of the methods includes receiving first audio data corresponding to an utterance; obtaining a first transcription of the first audio data; receiving data indicating (i) a selection of one or more terms of the first transcription and (ii) one or more of replacement terms; determining that one or more of the replacement terms are classified as a correction of one or more of the selected terms; in response to determining that the one or more of the replacement terms are classified as a correction of the one or more of the selected terms, obtaining a first portion of the first audio data that corresponds to one or more terms of the first transcription; and using the first portion of the first audio data that is associated with the one or more terms of the first transcription to train an acoustic model for recognizing the one or more of the replacement terms.

Acoustic model training using corrected terms

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for predicting follow-up queries to an initial transcription of an utterance. In some implementations, one or more follow-up queries that are pre-associated with a transcription of an initial utterance of a user are identified. A new or modified language model in which a respective probability associated with one or more of the follow-up queries is increased with respect to an initial language model is obtained. Subsequent audio data corresponding to a subsequent utterance of the user is then received. The subsequent audio data is processed using the new or modified language model to generate a transcription of the subsequent utterance. The transcription of the subsequent utterance is then provided for output to the user.

Follow-up voice query prediction

Methods, including computer programs encoded on a computer storage medium, for collaborative language model biasing. In one aspect, a method includes: obtaining (i) one or more initial candidate transcriptions, and (ii) one or more terms that are associated with a context; selecting one or more of the terms that are associated with the context, and that (i) do not occur in the candidate transcriptions, and (ii) are indicated as phonetically similar to one or more terms that do occur in the initial candidate transcriptions; generating one or more additional candidate transcriptions based on the (i) initial candidate transcriptions, and (ii) the selected terms; and providing the one or more additional candidate transcriptions to an automated speech recognizer.

Contextual improvement of voice query recognition

Growing community of inventors

Hedingen, Switzerland

Martin Baeuml