Google Inc.

Facebook Technologies, LLC

Clarifai, Inc.

Meta Platforms Technologies, LLC

Lucent Technologies Inc.

Ctrl-Labs Corporation

Adam L Berenzweig

Matthew Donald Zeiler

Patrick Kaifosh

Daniel Z Wetmore

David Petrou

Hartwig Adam

Matthew John Bridges

Andrews-Junior Kimbembe

Teresa Ko

Mikkel Crone Koser

Alexei Masterov

Paul Chang

Lorenz Francis Huelsbergen

Christopher Anderson

Jeffrey Scott Seely

Jeremy Hylton

Jack Wright Menzel

Kumar Mayur Thakur

Jason Reisman

Adam Al-natsheh

Vinod Ramachandran Marur

Laramie Leavitt

Brendan D Burns

Addy Ngan

Alan Huan Du

Ron J Weiss

Jason Edward Weston

Alexandre Barachant

Tudor Giurgica-Tiron

Christopher Osborn

Keith Ito

Austin Ha

Christopher Yan

Samy Bengio

Jay Michael Ponte

Ryan Michael Rifkin

Douglas Eck

Thomas Chadwick Walters

Thomas Reardon

Sally A Goldman

Christian Kaiserlian

Steven Kober

Brett Jurman

Nitzan Bartov

Robert Cochran

Zaina Edelson

Naor Brown

Geremy A Heitz, Iii

Joshua Duyan

Zhuo Wang

Mason Remaley

Michael Astolfi

Scott Draves

Sarvar M Patel

Semyon B Mizikovsky

Robert J Rance

Donald J Youtkus

Steven Anthony Falco

Szeyin Lee

Paulo Jose Telo Coelho

Luke O'Connor

Winston Edward Pekrul

Qiushi Mao

Ran Rubin

Shaker Islam

Edith Comas

Ian Sebastian Murphy Bicking

Attila Maczak

Lana Awad

Arielle Susu-Mago

Jasmine Stone

Robert John DiMaiolo

Luke O'CONNOR

The various implementations described herein include methods and systems for using a multi-stage in-air hand gesture to activate user-interface interactions in a way that ensures low-false positive rates. In one aspect, a method includes, while a gating gesture is maintained, receiving a first indication of performance of an adjustment gesture of a first magnitude directed to a user interface object associated with a plurality of values. The method further includes, in response to receiving the first indication, adjusting the user interface object to have a first state after moving through some of the plurality of values based on the first magnitude. The method also includes, after receiving an indication of a release of the gating gesture, in response to receiving a second indication of performance of the adjustment gesture, forgoing adjusting the user interface object such that the user interface object continues to have the first state.

Multi-stage gestures detected based on neuromuscular-signal sensors of a wearable device to activate user-interface interactions with low-false positive rates, and systems and methods of use thereof

The various implementations described herein include methods and systems for producing and modifying text using neuromuscular-signal-sensing devices. In one aspect, a method includes causing the display of a plurality of text terms input by a user. Using data from one or more neuromuscular-signal sensors in communication with the wearable device, an in-air hand gesture performed by the user is detected while the text terms are displayed. In response to the in-air hand gesture, a text-modification mode is enabled that allows for modifying the text terms input by the user. A target term is identified and, while the text-modification mode is enabled, data about a voice input provided by the user for modifying the target term is received. The method further includes causing a modification to the target term in accordance with the voice input from the user.

Techniques for neuromuscular-signal-based detection of in-air hand gestures for text production and modification, and systems, wearable devices, and methods for using these techniques

The various implementations described herein include methods and systems for using hand gestures detected at a wearable device to navigate a user interface. An example method includes receiving, via one or more neuromuscular-signal sensors of a wrist-wearable device worn by a user, data generated during performance of an in-air wrist movement by a wrist of the user. The method also includes moving a point of focus on the user interface in accordance with the in-air wrist movement; and receiving, via the one or more neuromuscular-signal sensors, additional data generated during performance of an in-air gesture by the user. The method further includes determining that the in-air gesture is an execution gesture; and executing a command corresponding to the execution gesture.

Navigating a user interface using in-air gestures detected via neuromuscular-signal sensors of a wearable device, and systems and methods of use thereof

Implementations of the present disclosure include actions of receiving image data of an image capturing a scene, receiving data describing one or more entities determined from the scene, the one or more entities being determined from the scene, determining one or more actions based on the one or more entities, each action being provided at least partly based on search results from searching the one or more entities, and providing instructions to display an action interface comprising one or more action elements, each action element being to induce execution of a respective action, the action interface being displayed in a viewfinder.

Smart camera user interface

A system for browsing, searching and/or viewing video content includes at least one user device and a server computer operably connected to the at least one user device. The server computer includes at least one processor operably connected to an electronic storage device, and the at least one processor is programmed with computer program instructions that, when executed, cause the server computer to present a first video on a user interface to the at least one user device, wherein the user interface presents scenes of the first video and semantic labels associated with the scenes of the first video, and wherein the user interface further presents confidence parameters associated with the scenes of the first video and the semantic labels. The server computer also obtains, during presentation of a first scene of the first video, a selection of a semantic label from a user of the at least one user device, then causes, during the presentation of the first scene of the first video, a jump from the first scene to a second scene of the first video based on the selection of the semantic label, the second scene being associated with the selected semantic label, and the jump from the first scene to the second scene causing the second scene to be presented on the user interface, and then updates the presentation of the semantic labels and the confidence parameters based on the jump from the first scene to the second scene such that the updated presentation of the semantic labels and the confidence parameters on the user interface are associated with the second scene.

User interface for labeling, browsing, and searching semantic labels within video

Methods and systems for providing input to an augmented reality system or an extended reality system based, at least in part, on neuromuscular signals. The methods and systems comprise detecting, using one or more neuromuscular sensors arranged on one or more wearable devices, neuromuscular signals from a user; determining that a computerized system is in a mode configured to provide input including text to the augmented reality system; identifying based, at least in part, on the neuromuscular signals and/or information based on the neuromuscular signals, the input, wherein the input is further identified based, at least in part, on the mode; and providing the identified input to the augmented reality system.

Neuromuscular text entry, writing and drawing in augmented reality systems

The disclosed computer-implemented method may include presenting, via a user interface, a sensory cue, and receiving, from neuromuscular sensors of a wearable device, various neuromuscular signals generated by a user wearing the wearable device, where the user generates the neuromuscular signals in response to the sensory cue being presented to the user via the user interface. The method may also include interpreting the received neuromuscular signals as input commands with respect to the sensory cue provided by the user interface, such that the input commands initiate performance of specified tasks within the user interface. The method may also include performing the specified tasks within the user interface according to the interpreted input commands. Various other methods, systems, and computer-readable media are also disclosed.

Systems, methods, and interfaces for performing inputs based on neuromuscular control

The disclosed human computer interface (HCI) system may include (1) at least one processor, (2) a plurality of sensors that detect one or more neuromuscular signals from a forearm or wrist of a user, and (3) memory that stores (A) one or more trained inferential models that determine an amount of force associated with the one or more neuromuscular signals and (B) computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to (I) identify the amount of force determined by the one or more trained inferential models, (II) determine that the amount of force satisfies a threshold force value, and in accordance with the determination that the amount of force satisfies the threshold force value, (III) generate a first input command for the HCI system. Various other devices, systems, and methods are also disclosed.

Devices, systems, and methods for controlling computing devices via neuromuscular signals of users

Generating a playlist may include designating a seed track in an audio library; identifying audio tracks in the audio library having constructs that are within a range of a corresponding construct of the seed track, where the constructs for the audio tracks are derived from frequency representations of the audio tracks, and the corresponding construct for the seed track is derived from a frequency representation of the seed track; and generating the playlist using at least some of the audio tracks that were identified.

Generating a playlist

Methods and apparatus for enabling a user or third-party to select or adjust parameters of one or more statistical models used to generate a musculoskeletal representation. The method comprises providing as input to the statistical model(s), a plurality of neuromuscular signals recorded by a plurality of neuromuscular sensors during performance of at least one gesture by a user, wherein the at least one gesture is performed by the user while wearing a wearable device having the plurality of neuromuscular sensors arranged thereon, rendering at least one visual representation based on an output of the statistical model(s), and receiving user or third-party input to adjust parameters of the statistical model(s) based on the rendered at least one visual representation, the user input including a selection of a particular statistical model of the statistical model(s) and/or an adjustment of parameters associated with the particular statistical model.

User-controlled tuning of handstate representation model parameters

Systems and methods for browsing, searching, and/or viewing video with associated semantic labels via a user interface are presented herein. Presentation of a user interface may be effectuated. The user interface may be configured to display scenes of one or more videos and/or one or more semantic labels associated with the scenes. A first scene associated with a first video and a first semantic label associated with the first scene may be displayed in the user interface. Playback of the first video in the user interface may cause one or more changes in the display of semantic labels over time, such that responsive to playback of the first video including display of a second scene after the first scene, a second semantic label is displayed and the first semantic label is no longer displayed. The second semantic label may be associated with the second scene.

In certain embodiments, a neural network may be trained to associated context information with multimedia items. In some embodiments, context predictions for multimedia items may be obtained via a neural network. A first multimedia item and a first task related to a first context prediction for the first multimedia item may be presented on a user interface. A user response to the first task may be obtained via the user interface. Based on the user response to the first task, prediction feedback related to the first context prediction or the first multimedia item may be provided to the neural network to cause the neural network to be updated based on the prediction feedback.

User interface for context labeling of multimedia items

Systems and methods for using neuromuscular information to improve speech recognition. The system includes a plurality of neuromuscular sensors arranged on one or more wearable devices and configured to continuously record a plurality of neuromuscular signals from a user, at least one storage device configured to store one or more trained statistical models for determining text based on audio input and the plurality of neuromuscular signals, at least one input interface configured to receive the audio input, and at least one computer processor programmed to obtain the audio input and the plurality of neuromuscular signals, provide as input to the one or more trained statistical models, the audio input and the plurality of neuromuscular signals or signals derived from the plurality of neuromuscular signals, and determine based, at least in part, on an output of the one or more trained statistical models, the text.

Systems and methods for improved speech recognition using neuromuscular information

Methods and apparatus for calibrating performance of one or more statistical models used to generate a musculoskeletal representation. The method comprises controlling presentation of instructions via a user interface to instruct the user to perform the at least one gesture and updating at least one parameter of the one or more statistical models based, at least in part on a plurality of neuromuscular signals recorded by a plurality of neuromuscular sensors during performance of the at least one gesture by the user.

Calibration techniques for handstate representation modeling using neuromuscular signals

Growing community of inventors

New York, NY, United States of America

Adam L Berenzweig