Sonos, Inc.

Other

Connor Kristopher Smith

Kurt Thomas Soto

Charles Conor Sleith

John G Tolomei

Nick D'Amato

Matthew David Anderson

Betty Lee

Patrick Kevin McPherson

Roberto Maria Dizon

Fiede Schillmoeller

Paul Andrew Bates

Nicholas D'Amato

Hendrik David Gideonse, Xix

John Tolomei

Patrick Kevin Mcpherson

Systems and methods for maintaining voice assistant persistence across multiple network microphone devices are described. In one example, first and second NMDs each identify a wake word based on detected sound, and are each transitioned from an inactive state to an active state in which the NMD captures and transmits sound data over a network interface. The first NMD is selected over the second NMD to output a first response, and both NMDs remain in the active state to further capture and transmit sound data. After further capturing and transmitting of sound data, the second NMD is selected over the first NMD to output a second response. After a predetermined time, one or both of the NMDs are transitioned back to the inactive state. The selection of one NMD over another for outputting a response can be based at least in part on user location information.

Voice assistant persistence across multiple network microphone devices

In one aspect, a network microphone device includes a plurality of microphones and is configured to capture a voice input via the one or more microphones, detect a wake word in the voice input, transmit data associated with the voice input to one or more remote computing devices associated with a voice assistant service, and receive a response from the one or more remote computing devices, the response comprising a playback command based on the voice input. The network microphone device may be configured to obtain verification information characterizing the voice input and, based on the verification information indicating that the voice input was spoken by an unverified user, functionally disable the NMD from performing the playback command.

Voice verification for media playback

Systems and methods for determining and adapting to changes in microphone performance of playback devices are disclosed herein. In one example, an audio input is received at an array of individual microphones of a network microphone device. Output microphone signals are generated from each of the individual microphones based on the audio input. The output microphone signals are analyzed to detect a trigger event. After detecting the trigger event, the output microphone signals are compared to detect aberrant behavior of one or more of the microphones. Optionally, corrective actions can be taken or suggested based on the detection of aberrant behavior of one or more microphones.

Determining and adapting to changes in microphone performance of playback devices

A device, such as Network Microphone Device or a playback device, receives an indication of a track change associated with a playback queue output by a media playback system. In response, an input detection window is opened for a given time period. During the given time period the device is arranged to receive an input sound data stream representing sound detected by a microphone. The input sound data stream is analyzed for a plurality of command keywords and/or a wake-word for a Voice Assistant Service (VAS) and, based on the analysis, it is determined that the input sound data stream includes voice input data comprising a command keyword or a wake-word for a VAS. In response, the device takes appropriate action such as causing the media playback system to perform a command corresponding to the command keyword or sending at least part of the input sound data stream to the VAS.

Command keywords with input detection windowing

As noted above, example techniques relate to offline voice control. A local voice input engine may process voice inputs locally when processing voice inputs via a cloud-based voice assistant service is not possible. Some techniques involve local (on-device) voice-assisted set-up of a cloud-based voice assistant service. Further example techniques involve local voice-assisted troubleshooting the cloud-based voice assistant service. Other techniques relate to interactions between local and cloud-based processing of voice inputs on a device that supports both local and cloud-based processing.

Offline voice control

As noted above, example techniques relate to toggling a cloud-based VAS between enabled and disabled modes. An example implementation involves a NMD detecting that the housing is in a first orientation and enabling a first mode. Enabling the first mode includes disabling voice input processing via a cloud-based VAS and enabling local voice input processing. In the first mode, the NMD captures sound data associated with a first voice input and detects, via a local natural language unit, that the first voice input comprises sound data matching one or more keywords. The NMD determines an intent of the first voice input and performs a first command according to the determined intent. The NMD may detect that the housing is in a second orientation and enables the second mode. Enabling the second mode includes enabling voice input processing via the cloud-based VAS.

VAS toggle based on device orientation

Systems and methods for generating training data are described herein. Pieces of metadata captured by a plurality of networked sensor systems can be captured, where each piece of metadata is associated with a specific set of sensor data captured by one of the plurality of networked sensor systems and includes a set of characteristics for the specific set of captured sensor data. A probabilistic model can be generated based on the received metadata and simulations can be performed based upon a training corpus by generating multiple scenarios, and, for each scenario, a scenario specific version of a particular annotated sample is generated by performing a simulation using the particular annotated sample. The scenario specific versions of annotated samples from the training corpus can be stored as a training data set on the at least one network device.

Systems and methods for generating labeled data to facilitate configuration of network microphone devices

Systems and methods for optimizing voice detection via a network microphone device are disclosed herein. In one example, individual microphones of a network microphone device detect sound. The sound data is captured in a first buffer and analyzed to detect a trigger event. Metadata associated with the sound data is captured in a second buffer and provided to at least one network device to determine at least one characteristic of the detected sound based on the metadata. The network device provides a response that includes an instruction, based on the determined characteristic, to modify at least one performance parameter of the NMD. The NMD then modifies the at least one performance parameter based on the instruction.

Voice detection optimization using sound metadata

In one aspect, a playback device includes a command-keyword engine having a local natural language unit (NLU). The playback device detects, via the command-keyword engine, a first command keyword in voice input of sound detected by one or more microphones of the playback device. The playback device determines whether the sound input data includes a keyword from a first predetermined library of keywords via a local natural language unit (NLU). The playback device transmits the input sound data to a second playback device over a local area network, the second playback device employing a second local NLU with a second predetermined library of keywords. The playback device receives a response from the second playback device and performs an action based on an intent determined by at least one of the first NLU or the second NLU according to the keywords in the voice input.

Locally distributed keyword detection

In one aspect, a playback device is provided that is configured to identify a trigger event indicating a request to associate the playback device with another playback device. Based on identifying the trigger event, the playback device is configured to create a first sound code based on a first sound specimen detected by the playback device. After identifying the trigger event, the playback device is configured to receive from the other playback device a sound object and based on receiving the sound object, identify a second sound code. The playback device is also configured to, based on the first sound code and the second sound code, determine that it and the other playback device have a spatial relationship. Based on that determination, the playback device is configured to cause it and the other playback device to be associated in accordance with the indicated request.

Networked devices, systems, and methods for associating playback devices based on sound codes

Systems and methods for distributed voice processing are disclosed herein. In one example, the method includes detecting sound via a microphone array of a first playback device and analyzing, via a first wake-word engine of the first playback device, the detected sound. The first playback device may transmit data associated with the detected sound to a second playback device over a local area network. A second wake-word engine of the second playback device may analyze the transmitted data associated with the detected sound. The method may further include identifying that the detected sound contains either a first wake word or a second wake word based on the analysis via the first and second wake-word engines, respectively. Based on the identification, sound data corresponding to the detected sound may be transmitted over a wide area network to a remote computing device associated with a particular voice assistant service.

Devices, systems, and methods for distributed voice processing

Systems and methods for optimizing voice detection via a network microphone device (NMD) based on a selected voice-assistant service (VAS) are disclosed herein. In one example, the NMD detects sound via individual microphones and selects a first VAS to communicate with the NMD. The NMD produces a first sound-data stream based on the detected sound using a spatial processor in a first configuration. Once the NMD determines that a second VAS is to be selected over the first VAS, the spatial processor assumes a second configuration for producing a second sound-data stream based on the detected sound. The second sound-data stream is then transmitted to one or more remote computing devices associated with the second VAS.

Voice detection optimization based on selected voice assistant service

A device, such as Network Microphone Device or a playback device, detecting an event associated with the device or a system comprising the device. In response, an input detection window is opened for a given time period. During the given time period the device is arranged to receive an input sound data stream representing sound detected by a microphone. The input sound data stream is analyzed for a plurality of keywords and/or a wake-word for a Voice Assistant Service (VAS) and, based on the analysis, it is determined that the input sound data stream includes voice input data comprising a keyword or a wake-word for a VAS. In response, the device takes appropriate action such as causing the media playback system to perform a command corresponding to the keyword or sending at least part of the input sound data stream to the VAS.

Input detection windowing

In one aspect, a network microphone device includes a plurality of microphones and is configured to detect sound via the one or more microphones. The network microphone device may capture sound data based on the detected sound in a first buffer, and capture metadata associated with the detected sound in a second buffer. The network microphone device may classify one or more noises in the detected sound and cause the network microphone device to perform an action based on the classification of the respective one or more noises.

Noise classification for event detection

In one aspect, a networked microphone device is configured to (i) receive sound data, (ii) determine, via the wake-word engine, that a first portion of the sound data is representative of a wake word, (iii) determine that a second networked microphone device was added to a media playback system, (iv) transmit the first portion of the sound data to a second networked microphone device, (v) begin determining a command to be performed by the first networked microphone device, (vi) receive an indication of whether the first portion of the sound data is representative of the wake word, and (vii) output a response indicative of whether the first portion of the sound data is representative of the wake word.

Growing community of inventors

New Hudson, MI, United States of America

Connor Kristopher Smith