Educational Testing Service

Keelan Evanini

Klaus Zechner

Yao Qian

Su-Youn Yoon

Lei Chen

Vikram Ramanarayanan

David Suendermann-Oeft

Xinhao Wang

Patrick Lange

Shasha Xie

Alexei V Ivanov

Derrick Higgins

Rutuja Ubale

Eugene Tsuprun

Je Hun Jeon

Isaac I Bejar

Jidong Tao

Jana Sukkarieh

Anastassia Loukina

Zhou Yu

Matthew Mulholland

Hillary Molloy

Robert A Pugh

James Bruno

Guangming Ling

Xie Sun

Wenting Xiong

Fei Huang

Miao Chen

Data is received that encapsulates a spoken response to a prompt text comprising a string of words. Thereafter, the received data is transcribed into a string of words. The string of words is then compared with a prompt so that a similarity grid representation of the comparison can be generated that characterizes a level of similarity between the string of words in the spoken response and the string of words in the prompt text. The grid representation is then scored using at least one machine learning model. The score indicates a likelihood of the spoken response having been off-topic. Data providing the encapsulated score can then be provided. Related apparatus, systems, techniques and articles are also described.

Detection of off-topic spoken responses using machine learning

Data is received that encapsulates a spoken response to a test question. Thereafter, the received data is transcribed into a string of words. The string of words is then compared with at least one source string so that a similarity grid representation of the comparison can be generated that characterizes a level of similarity between the string of words and the at least one source string. The grid representation is then scored using at least one machine learning model. The score indicates a likelihood of the spoken response having been plagiarized. Data providing the encapsulated score can then be provided. Related apparatus, systems, techniques and articles are also described.

Detection of plagiarized spoken responses using machine learning

Systems and methods are provided for conducting a simulated conversation with a language learner include determining a first dialog state of the simulated conversation. First audio data corresponding to simulated speech based on the dialog state is transmitted. Second audio data corresponding to a variable length utterance spoken in response to the simulated speech is received. A fixed dimension vector is generated based on the variable length utterance. A semantic label is predicted for the variable-length utterance based on the fixed dimension vector. A second dialog state of the simulated conversation is determined based on the semantic label, and third audio data corresponding to simulated speech is transmitted based on the second dialog state.

Exploring ASR-free end-to-end modeling to improve spoken language understanding in a cloud-based dialog system

Systems and methods are provided for automatic detection of plagiarized spoken responses during standardized testing of language proficiency. A first spoken response to a speaking task that elicits spontaneous speech and a second spoken response to a source-based speaking task that is assumed to be non-plagiarized are digitally recorded. A first and second sets of speaking proficiency features are calculated for the first and the second spoken response, respectively. The first spoken response is classifying as plagiarized or non-plagiarized based on the comparison between the first and the second set of speaking proficiency features. Corresponding apparatuses, systems, and methods are also disclosed.

Systems and methods for automatic detection of plagiarized spoken responses

Computer-implemented systems and methods are provided for automatically generating recitation items. For example, a computer performing the recitation item generation can receive one or more text sets that each includes one or more texts. The computer can determine a value for each text set using one or more metrics, such as a vocabulary difficulty metric, a syntactic complexity metric, a phoneme distribution metric, a phonetic difficulty metric, and a prosody distribution metric. Then the computer can select a final text set based on the value associated with each text set. The selected final text set can be used as the recitation items for a speaking assessment test.

Systems and methods for generating recitation items

Systems and methods for identifying a person's native language, are presented. A native language identification system, comprising a plurality of artificial neural networks, such as time delay deep neural networks, is provided. Respective artificial neural networks of the plurality of artificial neural networks are trained as universal background models, using separate native language and non-native language corpora. The artificial neural networks may be used to perform voice activity detection and to extract sufficient statistics from the respective language corpora. The artificial neural networks may use the sufficient statistics to estimate respective T-matrices, which may in turn be used to extract respective i-vectors. The artificial neural networks may use i-vectors to generate a multilayer perceptron model, which may be used to identify a person's native language, based on an utterance by the person in his or her non-native language.

Native language identification with time delay deep neural networks trained separately on native and non-native english corpora

Systems and methods are provided for implementing an educational dialog system. An initial task model is accessed that identifies a plurality of dialog states associated with a task, a language model configured to identify a response meaning associated with a received response, and a language understanding model configured to select a next dialog state based on the identified response meaning. The task is provided to a plurality of persons for training. The task model is updated by revising the language model and the language understanding model based on responses received to prompts of the provided task, and the updated task is provided to a student for development of speaking capabilities.

Computer-implemented systems and methods for a crowd source-bootstrapped spoken dialog system

Systems and methods are provided providing a spoken dialog system. Output is provided from a spoken dialog system that determines audio responses to a person based on recognized speech content from the person during a conversation between the person and the spoken dialog system. Video data associated with the person interacting with the spoken dialog system is received. A video engagement metric is derived from the video data, where the video engagement metric indicates a level of the person's engagement with the spoken dialog system.

Computer-implemented systems and methods for evaluating speech dialog system engagement via video

Systems and methods are provided for providing voice authentication of a candidate speaker. Training data sets are accessed, where each training data set comprises data associated with a training speech sample of a speaker and a plurality of speaker metrics, where the plurality of speaker metrics include a native language of the speaker. The training data sets are used to train a neural network, where the data associated with each training speech sample is a training input to the neural network, and each of the plurality of speaker metrics is a training output to the neural network. Data associated with a speech sample is provided to the neural network to generate a vector that contains values for the plurality of speaker metrics, and the values contained in the vector are compared to values contained in a reference vector associated with a known person to determine whether the candidate speaker is the known person.

Computer-implemented systems and methods for speaker recognition using a neural network

A method for scoring non-native speech includes receiving a speech sample spoken by a non-native speaker and performing automatic speech recognition and metric extraction on the speech sample to generate a transcript of the speech sample and a speech metric associated with the speech sample. The method further includes determining whether the speech sample is scorable or non-scorable based upon the transcript and speech metric, where the determination is based on an audio quality of the speech sample, an amount of speech of the speech sample, a degree to which the speech sample is off-topic, whether the speech sample includes speech from an incorrect language, or whether the speech sample includes plagiarized material. When the sample is determined to be non-scorable, an indication of non-scorability is associated with the speech sample. When the sample is determined to be scorable, the sample is provided to a scoring model for scoring.

Non-scorable response filters for speech scoring systems

Systems and methods are provided for generating an intelligibility score for speech of a non-native speaker. Words in a speech recording are identified using an automated speech recognizer, where the automated speech recognizer provides a string of words identified in the speech recording, and where the automated speech recognizer further provides an acoustic model likelihood score for each word in the string of words. For a particular word in the string of words, a context metric value is determined based upon a usage of the particular word within the string of words. An acoustic score for the particular word is determined based on the acoustic model likelihood score for the particular word from the automated speech recognizer. An intelligibility score is determined for the particular word based on the acoustic score for the particular word and the context metric value for the particular word.

Computer-implemented systems and methods for determining an intelligibility score for speech

Computer-implemented systems and methods are provided for assessing non-native speech proficiency. a non-native speech sample is processed to identify a plurality of vowel sound boundaries in the non-native speech sample. Portions of the non-native speech sample are analyzed within the vowel sound boundaries to extract vowel characteristics associated with a first vowel sound and a second vowel sound represented in the non-native speech sample. The vowel characteristics are processed to identify a first vowel pronunciation metric for the first vowel sound and a second vowel pronunciation metric for the second vowel sound, and the first vowel pronunciation metric and the second vowel pronunciation metric are processed to determine whether the non-native speech sample exhibits a distinction in pronunciation of the first vowel sound and the second vowel sound.

Systems and methods for an automated pronunciation assessment system for similar vowel pairs

Systems and methods are provided for automated detection of plagiarized spoken responses. A spoken response is processed to generate a text that is representative of the spoken response. The text is processed to remove disfluencies in the text and to identify a plurality of sentences in the text. A first numerical measure indicative of a number of words and phrases of the text that are included verbatim in a source text is determined. The source text has been designated as a source of plagiarized content. A second numerical measure indicative of an amount of the text that paraphrases portions of the source text is determined. A third numerical measure indicative of a similarity between sentences of the text and sentences of the source text is determined. A model is applied to the first, second, and third numerical measures to classify the spoken response as being plagiarized or non-plagiarized.

System and method for automated detection of plagiarized spoken responses

Computer-implemented systems and methods are provided for assessing non-native speech proficiency. A non-native speech sample is processed to identify a plurality of vowel sound boundaries in the non-native speech sample. Portions of the non-native speech sample are analyzed within the vowel sound boundaries to extract vowel characteristics. The vowel characteristics are used to identify a plurality of vowel space metrics for the non-native speech sample, and the vowel space metrics are used to determine a non-native speech proficiency score for the non-native speech sample.

Growing community of inventors

Pennington, NJ, United States of America

Keelan Evanini