Educational Testing Service

Modality.ai

Vikram Ramanarayanan

David Suendermann-Oeft

Yao Qian

Keelan Evanini

Patrick Lange

Alexei V Ivanov

Hardik Kothare

Michael Neumann

Lei Chen

Jidong Tao

Zhou Yu

Eugene Tsuprun

Klaus Zechner

Chee Wee Leong

Xinhao Wang

Gary Feng

Debanjan Ghosh

Robert A Pugh

Matthew Mulholland

Rutuja Ubale

Hillary Molloy

Saad Khan

Mark Katz

Eric Steinhauer

Ravindran Ramaswamy

Oliver Roesler

Andrew Cornish

David Pautler

Jackson Liscombe

Doug Habberstad

A system for identifying efficacious markers comprises memory storing collected multimodal digital markers from a first cohort experiencing a sign onset and a second cohort experiencing a non-sign onset. The system further comprises selection logic that identifies a subset of markers that best capture differences between the sign and non-sign onsets. The system further comprises responsiveness logic that determines a responsiveness parameter that specifies a rate of change in the subset. The system further comprises time to detect change logic that determines a time parameter that specifies a time required to detect change in the subset. The system further comprises sample size logic that determines how the responsiveness parameter and the time parameter change depending on sample size. The system further comprises sensitivity logic that determines a sensitivity parameter that specifies whether the subset detects condition deterioration during intervals when no changes are reported in an external standard.

Multimodal metrics for remote monitoring of condition progression

A computer-implemented method of generating interpretable, composite marker indexes that are discriminative and noise-robust is provided. The method comprises storing remotely collected multimodal digital markers from a first cohort and a second cohort. The method further comprises grouping multicollinear features in the multimodal digital markers into clusters, and then selecting representative features for the clusters for multiple classification tasks that require discrimination between the first cohort and the second cohort. The method further comprises linearly combining the representative features into an interpretable, composite marker index such that relative contributions of each of the representative features to the interpretable, composite marker index are known.

Interpretable index score for combining multimodal metrics for remote monitoring of condition progression

A computer-generated dialog session is customized for a user having a pathology characterized at least in part by a speech pathology. The user's speech is analyzed for spans of speech in which the starts and ends of the spans satisfy predetermined thresholds of time. Customization occurs by altering at least one of the following configurable parameters: (a) a threshold minimum signal strength of speech (dB) to consider as the start of the span of speech; (b) an adjustment factor by which signal strengths of background noise increases between consecutive spans of speech; (c) a threshold between signal strength during the span of speech and signal strength during the span of non-speech; (d) a start speech time threshold; and (e) an end speech time threshold.

Customizing computer generated dialog for different pathologies

Human-machine dialog is characterized by receiving data comprising a recording of an individual interacting with a dialog application simulating a conversation. Thereafter, the received data is parsed using automated speech recognition to result in text comprising a plurality of words. Features are extracted from the parsed data and then input an ensemble of different machine learning models each trained to generate a score characterizing a plurality of different dialog constructs. Thereafter, scores generated by the machine learning models for each of the dialog constructs are fused. A performance score is then generated based on the fused scores which characterizes a conversational proficiency of the individual interacting with the dialog application. Data can then be provided which includes or otherwise characterizes the generated score. Related apparatus, systems, techniques and articles are also described.

Ensemble-based machine learning characterization of human-machine dialog

Systems and methods for computer-implemented evaluation of a performance are provided. In a first aspect, a computer-implemented method of evaluating an interaction generates a first temporal record of first behavior features exhibited by a first entity during an interaction between a first entity and a second entity. A second temporal record is generated including second behavior features exhibited by a second entity during an interaction with a first entity. A determination is made that a first feature of a first temporal record is associated with a second feature of a second temporal record. The length of time that passes between the first feature and second feature is evaluated, and a determination is made that the length of time satisfies a temporal condition. A co-occurrence record associated with a first feature and a second feature is generated and included in a co-occurrence record data-structure.

Systems and methods for detecting co-occurrence of behavior in collaborative interactions

Systems and methods for identifying a person's native language and/or non-native language based on code-switched text and/or speech, are presented. The systems may be trained using various methods. For example, a language identification system may be trained using one or more code-switched corpora. Text and/or speech features may be extracted from the corpora and used, in combination with a per-word language identify of the text and/or speech, to train at least one machine learner. Code-switched text and/or speech may be received and processed by extracting text and/or speech features. These features may be fed into the at least one machine learner to identify the person's native language.

Automatic turn-level language identification for code-switched dialog

Systems and methods are provided for conducting a simulated conversation with a language learner include determining a first dialog state of the simulated conversation. First audio data corresponding to simulated speech based on the dialog state is transmitted. Second audio data corresponding to a variable length utterance spoken in response to the simulated speech is received. A fixed dimension vector is generated based on the variable length utterance. A semantic label is predicted for the variable-length utterance based on the fixed dimension vector. A second dialog state of the simulated conversation is determined based on the semantic label, and third audio data corresponding to simulated speech is transmitted based on the second dialog state.

Exploring ASR-free end-to-end modeling to improve spoken language understanding in a cloud-based dialog system

Systems and methods are provided for acquiring physical-world data indicative of interactions of a subject with an avatar for evaluation. An interactive avatar is provided for interaction with the subject. Speech from the subject to the avatar is captured, and automatic speech recognition is performed to determine content of the subject speech. Motion data from the subject interacting with the avatar is captured. A next action of the interactive avatar is determined based on the content of the subject speech or the motion data. The next action of the avatar is implemented, and a score for the subject is determined based on the content of the subject speech and the motion data.

Computer-implemented systems and methods for acquiring and assessing physical-world data indicative of avatar interactions

Systems and methods are provided for implementing an educational dialog system. An initial task model is accessed that identifies a plurality of dialog states associated with a task, a language model configured to identify a response meaning associated with a received response, and a language understanding model configured to select a next dialog state based on the identified response meaning. The task is provided to a plurality of persons for training. The task model is updated by revising the language model and the language understanding model based on responses received to prompts of the provided task, and the updated task is provided to a student for development of speaking capabilities.

Computer-implemented systems and methods for a crowd source-bootstrapped spoken dialog system

Systems and methods are provided providing a spoken dialog system. Output is provided from a spoken dialog system that determines audio responses to a person based on recognized speech content from the person during a conversation between the person and the spoken dialog system. Video data associated with the person interacting with the spoken dialog system is received. A video engagement metric is derived from the video data, where the video engagement metric indicates a level of the person's engagement with the spoken dialog system.

Computer-implemented systems and methods for evaluating speech dialog system engagement via video

Systems and methods are provided for a processor-implemented method of analyzing quality of sound acquired via a microphone. An input metric is extracted from a sound recording at each of a plurality of time intervals. The input metric is provided at each of the time intervals to a neural network that includes a memory component, where the neural network provides an output metric at each of the time intervals, where the output metric at a particular time interval is based on the input metric at a plurality of time intervals other than the particular time interval using the memory component of the neural network. The output metric is aggregated from each of the time intervals to generate a score indicative of the quality of the sound acquired via the microphone.

Processor-implemented systems and methods for determining sound quality

Computer-implemented systems and methods for evaluating a performance are provided. Motion of a user in a performance is detected using a motion capture device. Data collected by the motion capture device is processed with a processing system to identify occurrences of first and second types of actions by the user. The data collected by the motion capture device is processed with the processing system to determine values indicative of amounts of time between the occurrences. A non-verbal feature of the performance is determined based on the identified occurrences and the values. A score for the performance is generated using the processing system by applying a computer scoring model to the non-verbal feature.

Systems and methods for multi-modal performance scoring using time-series features

Systems and methods are provided for providing voice authentication of a candidate speaker. Training data sets are accessed, where each training data set comprises data associated with a training speech sample of a speaker and a plurality of speaker metrics, where the plurality of speaker metrics include a native language of the speaker. The training data sets are used to train a neural network, where the data associated with each training speech sample is a training input to the neural network, and each of the plurality of speaker metrics is a training output to the neural network. Data associated with a speech sample is provided to the neural network to generate a vector that contains values for the plurality of speaker metrics, and the values contained in the vector are compared to values contained in a reference vector associated with a known person to determine whether the candidate speaker is the known person.

Growing community of inventors

San Francisco, CA, United States of America

Vikram Ramanarayanan