Amazon Technologies, Inc.

Blackberry Corporation

Research in Motion Corporation

Computer Associates Think, Inc.

Eli Joshua Fidler

Genevieve Elizabeth Mak

Ariya Rastrow

Adam Chester Treat

David Francis Tapuska

Bjorn Hoffmeister

Fabian Andreas Bumberger

Roland Maximilian Rolf Maas

Matthew Nicholaos Staikos

Karl Arvid Nilsson

Nikko Strom

Aaron A Eakin

Michael Thomas Winkler

Maciej Makowski

Diamond Bishop

Sasitheran Shanmugarajah

Joseph Charles Mason

Sanjeev Mishra

Sabria Farheen

James G Droppo

Zoe Adams

Stanislaw Ignacy Pasko

Gengshen Fu

Andrew Smith

Nawdesh Uppal

Hans-Arno Jacobsen

Gautam Tiwari

Grant Strimel

Aaron Challenner

Veerdhawal Pande

Athanasios Mouchtaris

Andrew Oberlin

Nathan Anthony Susanj

Maciej Grabon

Michal Kowalczuk

Serguei Mankovski

Pavlo Stelmakh

Pavel Kveton

Reza Solgi

Sunwoo Kim

Markus Mueller

Benjamin Li

Dhruv Jain

Srikanth Subramaniam

Ashutosh Thakur

Ieliena Cannata

Muhammad Zain Mustafa

Maarten Van Segbroeck

Sree Hari Krishnan Parthasarathi

Pawel Zelazko

Cagdas Bak

Guoli Li

Alex Cheung

Pengcheng Wan

David Matheson

Antonio Gomes Araujo Netto

Antonio Gomes De Araujo Netto

The present disclosure provide techniques for processing natural language inputs on a user device or a system. In some embodiments, a device includes a component configured to predict an output(s) responsive to a user input, where the component learns the output based on historic processing performed by a more robust natural language processing system. The component processes ASR and NLU data to determine: (1) a natural language output that is presented as an audio or a visual output, (2) an action to be performed by a skill, and/or (3) a skill to respond to the user input. The component is automatically updated for new intents, features, business logic, etc., based on learning from the natural language processing system.

Natural language processing

Systems and methods for speech processing utilizing customized embeddings include receiving a first textual representation of first audio data and intent data indicating an intent of a first voice command. A first embedding may be generated from the textual representation and stored on the device. Second audio data representing a second voice command may be received and a second embedding may be generated therefrom. The first embedding may be determined to have at least a threshold similarity to the second embedding, and an intent may be determined that is associated with the second voice command. An action may be performed utilizing the intent.

Speech processing utilizing customized embeddings

A speech processing system that enables voice-privacy mode operations for resource limited computing devices. A device that has limited speech processing capabilities/is otherwise resource limited, connects to a local host. The user may then enable a voice-privacy mode for the limited device. When the mode is active, if the limited device detects an utterance, it sends the audio data of the utterance to the local host, which performs limited speech processing and sends the resulting data back to the limited device, which then sends it to a more powerful device to complete the speech processing.

Privacy mode for multi-device processing

A speech interface device is configured to detect an interrupt event and process a voice command without detecting a wakeword. The device includes on-device interrupt architecture configured to detect when device-directed speech is present and send audio data to a remote system for speech processing. This architecture includes an interrupt detector that detects an interrupt event (e.g., device-directed speech) with low latency, enabling the device to quickly lower a volume of output audio and/or perform other actions in response to a potential voice command. In addition, the architecture includes a device directed classifier that processes an entire utterance and corresponding semantic information and detects device-directed speech with high accuracy. Using the device directed classifier, the device may reject the interrupt event and increase a volume of the output audio or may accept the interrupt event, causing the output audio to end and performing speech processing on the audio data.

Device-directed utterance detection

A speech-processing system may determine potential endpoints in a user's speech. Such endpoint prediction may include determining a potential endpoint in a stream of audio data, and may additionally including determining an endpoint score representing a likelihood that the potential endpoint represents an end of speech representing a complete user input. When the potential endpoint has been determined, the system may publish a transcript of speech that preceded the potential endpoint, and send it to downstream components. The system may continue to transcribe audio data and determine additional potential endpoints while the downstream components process the transcript. The downstream components may determine whether the transcript is complete; e.g., represents the entirety of the user input. Final endpoint determinations may be made based on the results of the downstream processing including automatic speech recognition, natural language understanding, etc.

Endpointing in speech processing

Systems and methods for preemptive wakeword detection are disclosed. For example, a first part of a wakeword is detected from audio data representing a user utterance. When this occurs, on-device speech processing is initiated prior to when the entire wakeword is detected. When the entire wakeword is detected, results from the on-device speech processing and/or the audio data is sent to a speech processing system to determine a responsive action to be performed by the device. When the entire wakeword is not detected, on-device processing is canceled and the device refrains from sending the audio data to the speech processing system.

Preemptive wakeword detection

This disclosure proposes systems and methods enabling on-device/hybrid processing of speech requests using a hub device. The hub device is capable of receiving audio data from surrounding devices and performing speech processing on the audio data to improve latency and/or provide functionality to other devices within a private network. The hub device may receive multiple requests corresponding to different utterances. If the hub device receives a second utterance while processing a first utterance, the hub device may send an error notification, process the first utterance and the second utterance sequentially, suspend processing of the first utterance to process the second utterance first, send the second utterance to another hub device or remote system, or suspend processing of the first utterance and send the first utterance to the remote system in order to process the second utterance.

Speech processing for multiple inputs

Some speech processing systems may handle some commands on-device rather than sending the audio data to a second device or system for processing. The first device may have limited speech processing capabilities sufficient for handling common language and/or commands, while the second device (e.g., an edge device and/or a remote system) may call on additional language models, entity libraries, skill components, etc. to perform additional tasks. An intermediate data generator may facilitate dividing speech processing operations between devices by generating a stream of data that includes a first-pass ASR output (e.g., a word or sub-word lattice) and other characteristics of the audio data such as whisper detection, speaker identification, media signatures, etc. The second device can perform the additional processing using the data stream; e.g., without using the audio data. Thus, privacy may be enhanced by processing the audio data locally without sending it to other devices/systems.

Intermediate data for inter-device speech processing

Handling of touch events in a browser environment is disclosed. An example method includes, while a document is displayed on a touchscreen display of a device, detecting a touch event at the touchscreen display, and selectively processing the detected touch event using one of a default hander, a touch event handler, and a conversion to one or more mouse events, according to a touch event handling property defined for the document.

Handling of touch events in a browser environment

A device, system and method are provided for processing structured documents, such as webpages, for display. Various elements within the structured document are parsed and rendered by an electronic device for outputting to an external or integrated display. In response to a detected scaling instruction, such as a zoom in instruction, a selected region of the displayed document indicated by the instruction is scaled to a first scaled size, including any text content therein. Any text content contained within the region may be reflowed according to the bounds of its containing element or a viewport. A dominant alignment is determined from the element or elements contained within the selected region, and the portion of the scaled region to be output to the display is determined based on the dominant alignment.

Block zoom on a mobile electronic device

A method and computing device are provided for processing user events received via a user interface, such as a touchscreen, in multiple threads. When a user event is received for a target element in a webpage, the user event is dispatched to both a main browser thread and a secondary thread. The secondary thread processes user events in accordance with established default actions defined within the browser, while the main thread processes the user events in accordance with any event handlers defined for that target element. The main thread processing may be delayed by other interleaved task, and the secondary thread may be given priority over the main thread. When the secondary thread completes processing, an updated webpage is displayed. When the main thread subsequently completes processing, its updated rendering of the webpage is displayed. The secondary thread thus provides an early user interface response to the user event.

Processing user input events in a web browser

A device, system and method are provided for managing memory for rendering webpages and other structured documents that contain multiple regions. A backing store is created in memory for storing rendered document content. A main region of the structured document is rendered for display, divided into a set of tiles, and stored in the backing store. A subregion of the document is rendered and stored as tiles in the same backing store as well. At least a portion of the tiles for the main region and subregion intersecting with corresponding viewports are outputted to a display. When an active one of the viewports is changed and additional content of the document is to be rendered for display, tiles in the backing store used to store rendered but undisplayed data for the inactive viewport are released to store new rendered content for the active viewport.

Backing store memory management for rendering scrollable webpage subregions

Growing community of inventors

Toronto, Canada

Eli Joshua Fidler