Google Inc.

Gray Ice Higdon

Nicolo D'Ercole

Zaheed Sabur

Alvin Abdagic

Victor Carbune

Julia Proskurnia

Justin Lu

Behshad Behzadi

Megan Knight

Andrea Terwisscha Van Scheltinga

Luv Kothari

Bibo Xu

Jan Lamecki

Jacopo Sannazzaro Natta

Matthew Nirvan Sharifi

Michael Golikov

Bo Zhang

Mehek Sharma

Adrien Olczak

Ondrej Skopek

Daniel Valcarce

Mohamad Hassan Rom

Krishna Sapkota

Magali Boizot-Roche

Márius Sajgalík

Viesturs Zarins

Yang Lu

Akshay Kannan

Alessandro Agostini

Kevin I Kilgour

Jennifer Daniel

Srikanth Pandiri

Kevin Kilgour

Swante Scholz

Cristian Udrescu

Keqian Yan

Jackie Xiang

Jiawei Chen

Ajay Gokhale

Alexander Robertson

Dror Ayalon

Krzysztof Andrzej Goj

Shumin Zhai

Implementations described herein relate to an application and/or automated assistant that can identify arrangement operations to perform for arranging text during speech-to-text operations—without a user having to expressly identify the arrangement operations. In some instances, a user that is dictating a document (e.g., an email, a text message, etc.) can provide a spoken utterance to an application in order to incorporate textual content. However, in some of these instances, certain corresponding arrangements are needed for the textual content in the document. The textual content that is derived from the spoken utterance can be arranged by the application based on an intent, vocalization features, and/or contextual features associated with the spoken utterance and/or a type of the application associated with the document, without the user expressly identifying the corresponding arrangements. In this way, the application can infer content arrangement operations from a spoken utterance that only specifies the textual content.

Arranging and/or clearing speech-to-text content without a user providing express instructions

Implementations described herein relate to causing emoji(s) that are associated with a given emotion class expressed by a spoken utterance to be visually rendered for presentation to a user at a display of a client device of the user. Processor(s) of the client device may receive audio data that captures the spoken utterance, process the audio data to generate textual data that is predicted to correspond to the spoken utterance, and cause a transcription of the textual data to be visually rendered for presentation to the user via the display. Further, the processor(s) may determine, based on processing the textual data, whether the spoken utterance expresses a given emotion class. In response to determining that the spoken utterance expresses the given emotion class, the processor(s) may cause emoji(s) that are stored in association with the given emotion class to be visually rendered for presentation to the user via the display.

System(s) and method(s) for causing contextually relevant emoji(s) to be visually rendered for presentation to user(s) in smart dictation

Implementations described herein generally relate to generating a modification selectable element that may be provided for presentation to a user in a smart dictation session with an automated assistant. The modification selectable element may, when selected, cause a transcription, that includes textual data generated based on processing audio data that captures a spoken utterance and that is automatically arranged, to be modified. The transcription may be automatically arranged to include spacing, punctuation, capitalization, indentations, paragraph breaks, and/or other arrangement operations that are not specified by the user in providing the spoken utterance. Accordingly, a subsequent selection of the modification selectable element may cause these automatic arrangement operation(s), and/or the textual data locationally proximate to these automatic arrangement operation(s), to be modified. Implementations described herein also relate to generating the transcription and/or the modification selectable element on behalf of a third-party software application.

System(s) and method(s) to enable modification of an automatically arranged transcription in smart dictation

Determining whether, upon cessation of a second automated assistant session that interrupted and supplanted a prior first automated assistant session: (1) to automatically resume the prior first automated assistant session, or (2) to transition to an alternative automated assistant state in which the prior first session is not automatically resumed. Implementations further relate to selectively causing, based on the determining and upon cessation of the second automated assistant session, either the automatic resumption of the prior first automated assistant session that was interrupted, or the transition to the state in which the first session is not automatically resumed.

Determining whether to automatically resume first automated assistant session upon cessation of interrupting second session

Systems and methods described herein relate to determining whether to incorporate recognized text, that corresponds to a spoken utterance of a user of a client device, into a transcription displayed at the client device, or to cause an assistant command, that is associated with the transcription and that is based on the recognized text, to be performed by an automated assistant implemented by the client device. The spoken utterance is received during a dictation session between the user and the automated assistant. Implementations can process, using automatic speech recognition model(s), audio data that captures the spoken utterance to generate the recognized text. Further, implementations can determine whether to incorporate the recognized text into the transcription or cause the assistant command to be performed based on touch input being directed to the transcription, a state of the transcription, and/or audio-based characteristic(s) of the spoken utterance.

Voice commands for an automated assistant utilized in smart dictation

Some implementations process, using warm word model(s), a stream of audio data to determine a portion of the audio data that corresponds to particular word(s) and/or phrase(s) (e.g., a warm word) associated with an assistant command, process, using an automatic speech recognition (ASR) model, a preamble portion of the audio data (e.g., that precedes the warm word) and/or a postamble portion of the audio data (e.g., that follows the warm word) to generate ASR output, and determine, based on processing the ASR output, whether a user intended the assistant command to be performed. Additional or alternative implementations can process the stream of audio data using a speaker identification (SID) model to determine whether the audio data is sufficient to identify the user that provided a spoken utterance captured in the stream of audio data, and determine if that user is authorized to cause performance of the assistant command.

Contextual suppression of assistant command(s)

Growing community of inventors

Oberrieden, Switzerland

Nicolo D'Ercole