Amazon Technologies, Inc.

Other

Stanislaw Ignacy Pasko

Rohan Mutagi

Adam Barry Fineberg

Roger Robert Webster

Mark Aran Aiken

Olusanya Temitope Soyannwo

Maciej Makowski

Philip Alexander Lee

Makarand Damle

Marcin Fuszara

Vibhav Hemant Salgaonkar

Rui Liu

Ariya Rastrow

Eli Joshua Fidler

Bruno Dufour

Michal Kowalczuk

Andrew Oberlin

Michal Papierski

Aliaksei Kuzmin

Pawel Zelazko

Cagdas Bak

Ross William McGowan

Peipei Tan

Dmitry M Sharygin

Alexander Ivchenko

Some speech processing systems may handle some commands on-device rather than sending the audio data to a second device or system for processing. The first device may have basic speech processing capabilities sufficient for handling basic language and/or commands, while the second device (e.g., an edge device and/or a remote system) may call on additional language models, entity libraries, skill components, etc. to perform additional tasks. An intermediate data generator may facilitate dividing speech processing operations between devices by generating a stream of data that includes a first-pass ASR output (e.g., a word or sub-word lattice) and other characteristics of the audio data such as whisper detection, speaker identification, media signatures, etc. The second device can perform the additional processing using the data stream; e.g., without using the audio data. Thus, privacy may be enhanced by processing the audio data locally without sending it to other devices/systems.

Scheduling neural inference tasks

A speech interface device is configured to receive response data from a remote speech processing system for responding to user speech. This response data may be enhanced with information such as a remote ASR result(s) and a remote NLU result(s). The response data from the remote speech processing system may include one or more cacheable status indicators associated with the NLU result(s) and/or remote directive data, which indicate whether the remote NLU result(s) and/or the remote directive data are individually cacheable. A caching component of the speech interface device allows for caching at least some of this cacheable remote speech processing information, and using the cached information locally on the speech interface device when responding to user speech in the future. This allows for responding to user speech, even when the speech interface device is unable to communicate with a remote speech processing system over a wide area network.

Speech interface device with caching component

A speech interface device is configured with 'hybrid' capabilities, which allows the speech interface device to perform actions in response to user speech, even when the speech interface device is unable to communicate with a remote system over a wide area network (e.g., the Internet). A hybrid request selector of the speech interface device sends audio data representing user speech to both a remote speech processing system and a local speech processing component executing on the speech interface device, and then waits for a response from either or both components. The local speech processing component may start execution based on the audio data and subsequently suspend the execution until further instruction from the hybrid request selector. The hybrid request selector can then determine which response to use, and, depending on which response is chosen, may instruct the local speech processing component to either continue or terminate the suspended execution.

Hybrid speech interface device

A device can perform device arbitration, even when the device is unable to communicate with a remote system over a wide area network (e.g., the Internet). Upon detecting a wakeword in an utterance, the device can wait a period of time for data to arrive at the device, which, if received, indicates to the device that another speech interface device in the environment detected an utterance. If the device receives data prior to the period of time lapsing, the device can determine the earliest-occurring wakeword based on multiple wakeword occurrence times, and may designate whichever device that detected the wakeword first as the designated device to perform an action with respect to the user speech. To account for differences in sound capture latency between speech interface devices, a pre-calculated time offset value can be applied to wakeword occurrence time(s) during device arbitration.

Device arbitration by multiple speech processing systems

Some natural language command processing systems may handle some commands on a user device rather than sending input to another system for processing. Such a system may include an arbitration component for arbitrating between device and/or system processing. The arbitration component may execute in the system and render a device-specific decision as to whether the device will be able to process the input and/or execute the command, based on information known to the system about the device's capabilities. If the arbitration component predicts that the device will not be able to execute the command, the system may execute the command without waiting for a signal from the device. If the arbitration component predicts that the device will be able to execute the command, the system may halt processing to prevent duplicate execution.

Predicting on-device command execution

A hub is configured to provide voice control without assistance from a remote system, which allows the hub to provide a user with the ability to control second devices in an environment by issuing voice commands, even when the hub is unable to communicate with the remote system over a wide area network (e.g., the Internet). The hub is also configured to execute rules without assistance from the remote system, which allows the hub to execute rules, even when the hub is unable to communicate with the remote system over a wide area network (e.g., the Internet).

Architecture for a hub configured to control a second device while a connection to a remote system is unavailable

Some speech processing systems may handle some commands on-device rather than sending the audio data to a second device or system for processing. The first device may have limited speech processing capabilities sufficient for handling common language and/or commands, while the second device (e.g., an edge device and/or a remote system) may call on additional language models, entity libraries, skill components, etc. to perform additional tasks. An intermediate data generator may facilitate dividing speech processing operations between devices by generating a stream of data that includes a first-pass ASR output (e.g., a word or sub-word lattice) and other characteristics of the audio data such as whisper detection, speaker identification, media signatures, etc. The second device can perform the additional processing using the data stream; e.g., without using the audio data. Thus, privacy may be enhanced by processing the audio data locally without sending it to other devices/systems.

Intermediate data for inter-device speech processing

A speech interface device is configured to perform natural language understanding (NLU) processing in a manner that optimizes the use of resources on the speech interface device. In an example process, a domain classifier(s) is used to generate domain classifier scores associated with multiple candidate domains, and the candidate domains can then be evaluated, one candidate domain at a time, in accordance with the domain classifier scores (e.g., starting with a highest scoring candidate domain). For each candidate domain undergoing the evaluation, input data is by that domain's NLU model(s), and, as soon as a domain-specific NLU model(s) produces a NLU result with a confidence score that satisfies a threshold confidence score, the evaluation can be stopped for any remaining candidate domains.

Utilization of natural language understanding (NLU) models

Growing community of inventors

Gdansk, Poland

Stanislaw Ignacy Pasko