Snap Inc.

Assif Ziv

Meir Meshulam

Yaron Laufer

Elizabeth Izakson

Marcos Jimenez

Mark Kliger

Methods and systems are disclosed for iteratively training a user and a ML model to produce accurate inner speech outputs. The methods and systems access a ML model and perform a first training iteration in which EMG data corresponding to inner speech is processed by the machine learning model to decode the EMG data into a set of predicted phonemes, phoneme sounds, words or phrases. The methods and systems present the set of predicted phonemes, phoneme sounds, words or phrases to the user and form a first set of training data comprising the set of predicted phonemes, phoneme sounds, words or phrases, the EMG data, and the set of specified phonemes, phoneme sounds, words or phrases as ground truth information. The methods and systems update parameters of the ML model based on the first set of training data prior to starting a second training iteration.

Inner speech iterative learning loop

Methods and systems are disclosed for training users to produce inner speech. The system presents a prompt on a display device with one or more instructions for a user to produce inner speech and, in response to presenting the prompt, monitors EMG data to detect existence of inner speech. The system, in response to detecting existence of the inner speech, presents on the display device an indication that inner speech has been detected in the EMG data.

Training users to produce inner speech

Methods and systems are disclosed for training a machine learning (ML) model to detect inner speech. The system collects, by an electromyograph (EMG) communication device used by a user, a first set of EMG signals over a first time interval. The system generates a first plurality of features based on the first set of EMG signals and generates a first probability associated with presence of inner speech by processing the first plurality of features with a machine learning (ML) model. The system compares the first probability generated by the ML model to a specified threshold and detects presence of the inner speech of the user in response to determining that the first probability generated by the ML model transgresses the specified threshold.

EMG speech signal detection

Systems and methods are provided for performing operations comprising: detecting, by a plurality of electromyograph (EMG) electrodes of an EMG communication device, subthreshold muscle activation signals of one or more muscles associated with speech production, the subthreshold muscle activation signals being generated in response to inner speech of a user; applying a transform to the subthreshold muscle activation signals detected by the plurality of EMG electrodes to represent the subthreshold muscle activation signals as a plurality of maximally independent components; accessing an interference template corresponding to an eye blink component; filtering the plurality of maximally independent components based on the interference template to provide filtered maximally independent components; and regenerating a filtered version of the subthreshold muscle activation signals based on the filtered maximally independent components.

Removing eye blinks from EMG speech signals

Systems and methods are provided for performing operations comprising: detecting, by one or more electromyograph (EMG) electrodes of an EMG communication device, subthreshold muscle activation signals of one or more muscles associated with speech production, the subthreshold muscle activation signals being generated in response to inner speech of a user; applying a machine learning technique to the subthreshold muscle activation signals to estimate one or more speech features corresponding to the subthreshold muscle activation signals, the machine learning technique being trained to establish a relationship between a plurality of training subthreshold muscle activation signals and ground truth speech features; generating visual or audible output based on the one or more speech features; and causing the visual or audible output to be processed by a messaging application to engage a feature of the messaging application.

Growing community of inventors

Beit Yitzhak-Sha'ar Hefer, Israel

Assif Ziv