Facebook Technologies, LLC

Ctrl-labs Corporation

Meta Platforms Technologies, LLC

Patrick Kaifosh

Adam L Berenzweig

Thomas Reardon

Daniel Z Wetmore

Alexandre Barachant

Tudor Giurgica-Tiron

Adam Al-natsheh

Erik Schomburg

Timothy Machado

Steven Kober

Jeffrey Scott Seely

Alan Huan Du

Brett Jurman

Christopher Osborn

Qiushi Mao

Zhuo Wang

Jason Reisman

Nitzan Bartov

Robert Cochran

Joshua Merel

Steven Brian Demers

Naor Brown

Joshua Duyan

Nadine Hussami

Jasmine Stone

Robert John DiMaiolo

Calvin Tong

Anthony D Moschella

Michael Isaac Mandel

Steven Demers

Tim Machado

A system comprising a plurality of neuromuscular sensors, each of which is configured to record a time-series of neuromuscular signals from a surface of a user's body; and at least one computer hardware processor programmed to perform: applying a source separation technique to the time series of neuromuscular signals recorded by the plurality of neuromuscular sensors to obtain a plurality of neuromuscular source signals and corresponding mixing information; providing features, obtained from the plurality of neuromuscular source signals and/or the corresponding mixing information, as input to a trained statistical classifier and obtaining corresponding output; and identifying, based on the output of the trained statistical classifier, and for each of one or more of the plurality of neuromuscular source signals, an associated set of one or more biological structures.

Systems and methods for identifying biological structures associated with neuromuscular source signals

Methods and apparatus for enabling a user or third-party to select or adjust parameters of one or more statistical models used to generate a musculoskeletal representation. The method comprises providing as input to the statistical model(s), a plurality of neuromuscular signals recorded by a plurality of neuromuscular sensors during performance of at least one gesture by a user, wherein the at least one gesture is performed by the user while wearing a wearable device having the plurality of neuromuscular sensors arranged thereon, rendering at least one visual representation based on an output of the statistical model(s), and receiving user or third-party input to adjust parameters of the statistical model(s) based on the rendered at least one visual representation, the user input including a selection of a particular statistical model of the statistical model(s) and/or an adjustment of parameters associated with the particular statistical model.

User-controlled tuning of handstate representation model parameters

A method for determining spatial information for a multi-segment articulated rigid body system having at least an anchored segment and a non-anchored segment coupled to the anchored segment, each segment in the multi-segment articulated rigid body system representing a respective body part of a user, the method comprising: obtaining signals recorded by a first autonomous movement sensor coupled to a body part of the user represented by the non-anchored segment; providing the obtained signals as input to a trained statistical model and obtaining corresponding output of the trained statistical model; and determining, based on the corresponding output of the trained statistical model, spatial information for at least the non-anchored segment of the multi-segment articulated rigid body system. Determining the spatial information may include determining the position and/or orientation of the non-anchored segment relative to the anchor point and/or determining a spatial relationship between the anchored and non-anchored segments.

System and method for measuring the movements of articulated rigid bodies

The disclosed systems and methods are generally directed to interpreting neuromuscular signals. The system includes (A) a plurality of neuromuscular sensors that detect a plurality of neuromuscular signals from a user and (B) at least one multiplexer in communication with the plurality of neuromuscular sensors and that is capable of dynamically adjusting neuromuscular sensor processing based on neuromuscular signal characteristics. A computer processor is programmed to (i) receive a set of neuromuscular signals from the plurality of neuromuscular sensors, (ii) determine, via a real-time system, at least one signal characteristic included in a neuromuscular signal, where the neuromuscular signal is associated with a first neuromuscular sensor included in the plurality of neuromuscular sensors; and (iii) dynamically reconfigure the processing of neuromuscular signals from the plurality of neuromuscular sensors based on an output from the multiplexer. Various other methods, systems, and computer-readable media are also disclosed.

Systems and methods for mitigating neuromuscular signal artifacts

The disclosed method may include receiving neuromuscular activity data over a first time series from a first sensor on a wearable device donned by a user receiving ground truth data over a second time series from a second sensor that indicates a body part state of a body part of the user, generating one or more training datasets by time-shifting at least a portion of the neuromuscular activity data over the first time series relative to the second time series, to associate the neuromuscular activity data with at least a portion of the ground truth data, and training one or more inferential models based on the one or more training datasets. Various other related methods and systems are also disclosed.

Methods and apparatuses for low latency body state prediction based on neuromuscular data

Systems and methods for using neuromuscular information to improve speech recognition. The system includes a plurality of neuromuscular sensors arranged on one or more wearable devices and configured to continuously record a plurality of neuromuscular signals from a user, at least one storage device configured to store one or more trained statistical models for determining text based on audio input and the plurality of neuromuscular signals, at least one input interface configured to receive the audio input, and at least one computer processor programmed to obtain the audio input and the plurality of neuromuscular signals, provide as input to the one or more trained statistical models, the audio input and the plurality of neuromuscular signals or signals derived from the plurality of neuromuscular signals, and determine based, at least in part, on an output of the one or more trained statistical models, the text.

Systems and methods for improved speech recognition using neuromuscular information

Methods and apparatus for substantially real-time detection of spike events in neuromuscular data. The method comprises receiving a plurality of neuromuscular signals from a plurality of neuromuscular sensors arranged on one or more wearable devices worn by a user, detecting, based on the plurality of neuromuscular signals or information derived from the plurality of neuromuscular signals, at least one spike event corresponding to firing of an action potential in at least one motor unit, determining, based on the plurality of neuromuscular signals or the information derived from the plurality of neuromuscular signals, a biological source of the detected at least one spike event, and generating at least one output based, at least in part, on the detected at least one spike event and/or the determined biological source of the detected at least one spike event.

Real-time spike detection and identification

Methods and apparatus for calibrating performance of one or more statistical models used to generate a musculoskeletal representation. The method comprises controlling presentation of instructions via a user interface to instruct the user to perform the at least one gesture and updating at least one parameter of the one or more statistical models based, at least in part on a plurality of neuromuscular signals recorded by a plurality of neuromuscular sensors during performance of the at least one gesture by the user.

Calibration techniques for handstate representation modeling using neuromuscular signals

Systems and methods for using neuromuscular information to improve speech recognition. The system includes a plurality of neuromuscular sensors, arranged on one or more wearable devices, wherein the plurality of neuromuscular sensors is configured to continuously record a plurality of neuromuscular signals from a user, at least one storage device configured to store one or more trained statistical models, and at least one computer processor programmed to provide as an input to the one or more trained statistical models, the plurality of neuromuscular signals or signals derived from the plurality of neuromuscular signals, determine based, at least in part, on an output of the one or more trained statistical models, at least one instruction for modifying an operation of a speech recognizer, and provide the at least one instruction to the speech recognizer.

Wearable devices and methods for improved speech recognition

Methods and apparatus for adapting a control mapping associating sensor signals with control signals for controlling an operation of a device. The method comprises obtaining first state information for an operation of the device, providing the first state information as input to an intention model associated with an operation of the device and obtaining corresponding first intention model output, providing a plurality of neuromuscular signals recorded from a user and/or signals derived from the neuromuscular signals as inputs to a first control mapping and obtaining corresponding first control mapping output, and updating the first control mapping using the inputs provided to the first control mapping and the first intention model output to obtain a second control mapping.

Adaptive system for deriving control signals from measurements of neuromuscular activity

Methods and apparatus for providing a dynamically-updated computerized musculo-skeletal representation comprising a plurality of rigid body segments connected by joints. The method comprises recording, using a plurality of autonomous sensors arranged on one or more wearable devices, a plurality of autonomous signals from a user, wherein the plurality of autonomous sensors include a plurality of neuromuscular sensors configured to record neuromuscular signals. The method further comprises providing as input to a trained statistical model, the plurality of neuromuscular signals and/or information based on the plurality of neuromuscular signals. The method further comprises determining, based on an output of the trained statistical model, musculo-skeletal position information describing a spatial relationship between two or more connected segments of the plurality of rigid body segments of the computerized musculo-skeletal representation, and updating the computerized musculo-skeletal representation based, at least in part, on the musculo-skeletal position information.

Methods and apparatus for predicting musculo-skeletal position information using wearable autonomous sensors

Methods and apparatus for anonymizing neuromuscular signals used to generate a musculoskeletal representation. The method comprises recording, using a plurality of neuromuscular sensors arranged on one or more wearable devices, a plurality of neuromuscular signals from a user, providing as input to a trained statistical model, the plurality of neuromuscular signals and/or information based on the plurality of neuromuscular signals; and generating, the musculoskeletal representation based, at least in part, on an output of the trained statistical model, wherein the musculoskeletal representation is an anonymized musculoskeletal representation from which at least one personal characteristic of the user has been removed.

Techniques for anonymizing neuromuscular signal data

Methods and apparatus for controlling a physical object in an environment based, at least in part, on neuromuscular signals. The method comprises recording a plurality of neuromuscular signals from a plurality of neuromuscular sensors arranged on one or more wearable devices worn by a user, receiving a selection of a physical object within the environment, and controlling, based at least in part on the plurality of neuromuscular signals and/or information based on the plurality of neuromuscular signals, an operation of the selected object within the environment.

Neuromuscular control of physical objects in an environment

Computerized systems, methods, and computer-readable storage media storing code for implementing the methods are described for providing dynamically-updated musculoskeletal information. One such system includes a processor is programmed to: provide, as an input to a trained inference model, information based on a plurality of neuromuscular signals from a user and information based on at least one image of the user; determine, based on an output of the trained inference model, position information describing a spatial relationship between two or more connected musculoskeletal segments of the user and/or force information describing a force exerted by at least one musculoskeletal segment of the user; and output the position information and/or the force information.

Camera-guided interpretation of neuromuscular signals

Computerized systems, methods, and computer-readable storage media storing code for implementing the methods are provided for training an inference model used to generate a musculoskeletal representation. One such system includes a processor programmed to: determine, based on information obtained from at least one image, position information describing a spatial relationship between two or more connected musculoskeletal segments of a user; determine, based on a plurality of neuromuscular signals, force information; associate the position information with the force information; train an inference model to output a musculoskeletal representation consistent with the position information and/or the force information when neuromuscular input signals provided to the inference model have at least one predetermined characteristic, to produce an updated inference model; and cause the updated inference model to be stored in a memory.

Growing community of inventors

New York, NY, United States of America

Patrick Kaifosh