Orcam Technologies Ltd.

Qcue, Inc.

Yonatan Wexler

Amnon Shashua

Avi Barliya

Aviad Maizels

Roi Nathan

Tal Rosenwein

Oren Tadmor

Dave Bortz

Ronen Katsir

Erez Na'aman

Nir Sancho

Yair Deitcher

Roy Weiss

Tai Rosenwein

Doron Weizman

Systems and methods are disclosed for determining an intent to speak based on minute facial skin movements. In one implementation, a system may include a processor configured to control at least one coherent light source to illuminate a region of a face. The processor may receive from at least one sensor, reflection signals indicative of coherent light reflected from the face. The reflection signals may be analyzed to determine minute facial skin movements associated with silent speech. Then, based on the determined minute facial skin movements associated with the silent speech, the processor may determine a speech intent, and initiate, prior to an audible utterance of the silent speech, an action based on the determined speech intent.

Initiating an action based on a detected intention to speak

Systems and methods are disclosed for speech detection from facial skin movements. In one implementation, a system may include at least one light source, at least one sensor configured to receive light reflections from the at least one light source; and a processor configured to control the at least one light source to illuminate a region of a face of a user in a time interval. The processor may receive reflection signals indicative of light reflected from the face in the time interval. The reflection signals may be analyzed to determine facial skin movements in the time interval, wherein determining the sequence of words includes using an artificial neural network and a motion-to-language analysis. Then, based on the determined facial skin movements, the processor may determine a sequence of words associated with the facial skin movements, and output a textual transcription corresponding with the determined sequence of words.

Speech detection from facial skin movements

A multifunctional earpiece comprising an ear-mountable housing, a speaker integrated with the ear-mountable housing for presenting sound, a light source integrated with the ear-mountable housing for projecting light toward skin of the wearer's face, a light detector integrated with the ear-mountable housing and configured to receive reflections from the skin corresponding to facial skin micromovements indicative of prevocalized words of the wearer, and wherein the multifunctional earpiece is configured to simultaneously present the sound through the speaker, project the light toward the skin, and detect the received reflections indicative of the prevocalized words.

Earbud with facial micromovement detection capabilities

Head mountable systems, methods, and non-transitory computer readable media including instructions for identifying individuals using facial skin micromovements are disclosed. An example head mountable system may include a wearable housing, a coherent light source, a detector, and at least one processor. The at least one processor may analyze reflection signals from the detector to determine specific facial skin micromovements of an individual wearing the head mountable system. Thereafter, the at least one processor may access memory correlating a plurality of facial skin micromovements with the individual, and search for a match between the determined specific facial skin micromovements and at least one of the plurality of facial skin micromovements in the memory. If a match is identified, the processor may initiate a first action; and if a match is not identified, the at least one processor may initiate a second action different from the first action.

Using facial skin micromovements to identify a user

Systems, methods, and non-transitory computer readable media including instructions for interpreting facial skin micromovements are disclosed. An example head mountable system includes a housing configured to be worn on a head of a wearer. The head mountable system also includes at least one detector integrated with the housing and configured to receive light reflections from a facial region of the head and to output associated reflection signals. The head mountable system also includes at least one microphone associated with the housing and configured to capture sounds produced by the wearer and to output associated audio signals. The head mountable system also includes and at least one processor in the housing, configured to use both the reflection signals and the audio signals to generate output that corresponds with words articulated by the wearer.

Using a wearable to interpret facial skin micromovements

Systems, methods, and non-transitory computer readable media including instructions for noise suppression are described. A head mountable system for noise suppression includes a wearable housing; a coherent light source configured to project light towards a facial region of the head; a detector configured to receive coherent light reflections from the facial region associated with facial skin micromovements and to output associated reflection signals; and a processor configured to: analyze the reflection signals to determine speech timing; receive audio signals from at least one microphone; correlate the reflection signals with the received audio signals to determine portions of the audio signals associated with the words spoken by the wearer; and output the determined portions of the audio signals associated with words spoken by the wearer, while omitting output of other portions of the audio signals not containing the words spoken by the wearer.

Wearable for suppressing sound other than a wearer's voice

A system for generating processed audio signals may include an earphone comprising a microphone configured to capture sounds from an environment of the earphone, a first communication component, and a first processor; and an earphone case comprising an earphone compartment configured to receive the earphone, a second communication component, and a second processor. The first processor may be programmed to receive a captured audio signal representative of the sounds captured by the microphone; and transmit the captured audio signal via the first communication component. The second processor may be programmed to receive the captured audio signal via the second communication component; generate a processed audio signal based on analysis of the captured audio signal; and transmit the processed audio signal via the second communication component to the first communication component.

Earphone smartcase with audio processor

Systems, methods, and non-transitory computer readable media including instructions for interpreting facial skin micromovements are disclosed. Interpreting facial skin micromovements includes receiving during a first time period first signals representing prevocalization facial skin micromovements, and receiving during a second time period succeeding the first time period, second signals representing sounds. The sounds are analyzed to identify words spoken during the second time period, and the words are correlated with the prevocalization facial skin micromovements received during the first time period. The correlations are stored for future use. During a third time period, third signals representing facial skin micromovements are received in an absence of vocalization. Using the correlations, language associated with the third signals is identified and outputted.

Identifying silent speech using recorded speech

The presently disclosed embodiments may include a non-transitory computer readable medium containing instruction that when executed by at least one processor cause the at least one processor to perform thresholding operations for interpretation of facial skin micromovements. The operations may include detecting facial micromovements in an absence of perceptible vocalization associated with the facial micromovements, determining an intensity level of the facial micromovements and comparing the determined intensity level with a threshold. When the intensity level is above the threshold, the operations may include interpreting the facial micromovements and when the intensity level falls beneath the threshold, the operations may include disregarding the facial micromovements.

Threshold facial micromovement intensity triggers interpretation

Systems, methods, and non-transitory computer readable media including instructions for establishing nonvocalized conversations are disclosed. The operations include establishing a wireless communication channel for enabling a nonvocalized conversation via a first wearable device and a second wearable device. The operations also include detecting by the first wearable device first facial skin micromovements occurring in an absence of perceptible vocalization, and transmitting a first communication from the first wearable device to the second wearable device, wherein the first communication is derived from the first facial skin micromovements and is transmitted for presentation via the second wearable device. The operations also include receiving a second communication from the second wearable device, wherein the second communication is derived from second facial skin micromovements detected by the second wearable device, and presenting the second communication to a wearer of the first wearable device.

Facilitating silent conversation

Systems, methods, and computer program products are disclosed for removing noise from facial skin micromovement signals. Removing noise from facial skin micromovements includes, during a time period when an individual is involved in at least one non-speech-related physical activity, operating a light source in a manner enabling illumination of a facial skin region of the individual; receiving signals representing light reflections from the facial skin region; analyzing the received signals to identify a first reflection component indicative of prevocalization facial skin micromovements and a second reflection component associated with the at least one non-speech-related physical activity; and filtering out the second reflection component to enable interpretation of words from the first reflection component indicative of the prevocalization facial skin micromovements.

Interpreting words prior to vocalization

Some disclosed embodiments involve one or more of: receiving during one or more time periods, signals representing coherent light reflections associated with facial skin micromovements; analyzing the coherent light reflections to determine one or more pluralities of words associated with the facial skin micromovements; receiving information indicative of contextual conditions in which the facial skin micromovements occurred; accessing a plurality of control rules correlating a plurality of actions with a plurality of contextual conditions, wherein a first control rule prescribes a form of private presentation based on the first contextual condition, and a second control rule prescribes a form of non-private presentation based on the second contextual condition; upon receipt of the first information, implementing the first control rule to privately output a first plurality of words; and/or upon receipt of the second information, implementing the second control rule to non-privately output a second plurality of words.

Providing context-driven output based on facial micromovements

Systems, methods, and non-transitory computer readable media including instructions for determining and interpreting facial skin micromovements are disclosed. Determining facial skin micromovements includes controlling at least one coherent light source for projecting a plurality of light spots on a facial region of an individual, wherein the plurality of light spots includes at least a first light spot and a second light spot spaced from the first light spot. Reflected light from the first light spot is analyzed to determine changes in first spot reflections. Reflected light from the second light spot is analyzed to determine changes in second spot reflections. Based on the determined changes in the first spot reflections and the second spot reflections, the facial skin micromovements are determined. The facial skin micromovements derived from analyzing the first spot reflections and analyzing the second spot reflections are interpreted. An output of the interpretation is generated.

Using projected spots to determine facial micromovements

Systems, methods, and non-transitory computer-readable media including instructions for detecting and utilizing facial skin micromovements are disclosed. In some non-limiting embodiments, the detection of the facial skin micromovements occurs using a speech detection system that may include a wearable housing, a light source (either a coherent light source or a non-coherent light source), a light detector, and at least one processor. One or more processors may be configured to analyze light reflections received from a facial region to determine the facial skin micromovements, and extract meaning from the determined facial skin micromovements. Examples of meaning that may be extracted from the determined facial skin micromovements may include words spoken by the individual (either silently spoken or vocally spoken), an identification of the individual, an emotional state of the individual, a heart rate of the individual, a respiration rate of the individual, or any other biometric, emotion, or speech-related indicator.

Personal presentation of prevocalization to improve articulation

Systems, methods, and non-transitory computer readable media including instructions for performing operations for continuous authentication based on facial skin micromovements is disclosed. The operations may include receiving during an ongoing electronic transaction, first signals representing coherent light reflections associated with first facial skin micromovements during a first time period and second signals representing coherent light reflections associated with second facial skin micromovements during a second time period. The operations may also include determining, using the first and second signals, that a specific individual is associated with the first and second facial skin micromovements. The operations may also include receiving during the ongoing electronic transaction third signals representing coherent light reflections associated with third facial skin micromovements. The operations may further include determining, using the third signals, that the third facial skin micromovements are not associated with the specific individual, and initiating an action based on the determination.

Growing community of inventors

Jerusalem, Israel

Yonatan Wexler