Educational Testing Service

Isaac I Bejar

Klaus Zechner

Derrick Higgins

Sean J Whalen

Lei Chen

Keelan Evanini

Michael Flor

Su-Youn Yoon

Chee Wee Leong

Michael Heilman

Yoko Futagi

Keith S Reid-Green

Shasha Xie

Xiaoming Xi

Dennis C Quardt

Daniel Israel Zuckerman

Harriet Priscilla Trenholm

Peter D Brittingham

Robert L Rarich

Richard N Devore

Je Hun Jeon

David Michael Williamson

Michelle Paulette Martin

Joanna Gorin

Clark L Chalifour

Anne S Hone

Henry I Braun

Patrick Charles Kyllonen

Guangming Ling

Systems and methods described herein utilize supervised machine learning to generate a model for scoring interview responses. The system may access a training response, which in one embodiment is an audiovisual recording of a person responding to an interview question. The training response may have an assigned human-determined score. The system may extract at least one delivery feature and at least one content feature from the audiovisual recording of the training response, and use the extracted features and the human-determined score to train a response scoring model for scoring interview responses. The response scoring model may be configured based on the training to automatically assign scores to audiovisual recordings of interview responses. The scores for interview responses may be used by interviewers to assess candidates.

Systems and methods for assessing structured interview responses

Systems and methods are provided for determining a susceptibility of a computer-implemented automated scoring engine to gaming strategies. A plurality of responses to a prompt are provided to a computer-implemented automated scoring engine to receive a first set of scores. A first transformation is performed on each of the plurality of responses to generate a first set of transformed responses. The first set of transformed responses is provided to the computer-implemented automatic scoring engine to receive a second set of scores, and a gaming susceptibility metric is determined based on the first set of scores and the second set of scores.

Systems and methods for evaluating susceptibility of automated scoring engines to gaming behavior

A method for scoring non-native speech includes receiving a speech sample spoken by a non-native speaker and performing automatic speech recognition and metric extraction on the speech sample to generate a transcript of the speech sample and a speech metric associated with the speech sample. The method further includes determining whether the speech sample is scorable or non-scorable based upon the transcript and speech metric, where the determination is based on an audio quality of the speech sample, an amount of speech of the speech sample, a degree to which the speech sample is off-topic, whether the speech sample includes speech from an incorrect language, or whether the speech sample includes plagiarized material. When the sample is determined to be non-scorable, an indication of non-scorability is associated with the speech sample. When the sample is determined to be scorable, the sample is provided to a scoring model for scoring.

Non-scorable response filters for speech scoring systems

A method and system for automatically generating a scoring model for scoring a speech sample are disclosed. One or more training speech samples are received in response to a prompt. One or more speech features are determined for each of the training speech samples. A scoring model is then generated based on the speech features. At least one of the training speech samples may be a high entropy speech sample. An evaluation speech sample is received and a score is assigned to the evaluation speech sample using the scoring model. The evaluation speech sample may be a high entropy speech sample.

Method and system for the automatic generation of speech features for scoring high entropy speech

A method and system for providing immediate diagnostic feedback on speech samples of non-native speakers are disclosed. A scoring model is generated based on speech features extracted from one or more training speech samples. An evaluation speech sample is received and speech features of the evaluation speech sample are determined. Based on the scoring model and the speech features, diagnostic feedback is provided to the speaker. In an alternate embodiment, speech features are extracted from an evaluation speech sample. The speech features are then compared with optimal values, ranges of values, or norms for those features. Based on the result of the comparison, diagnostic feedback is provided to the speaker.

Method and system for using automatic generation of speech features to provide diagnostic feedback

The present invention provides systems and methods for controlled presentation of constructed responses through a variety of computer applications to human evaluators for assessment. The systems and methods further provide for controlled presentation of the constructed responses to minimize the influences of psychometric factors on the accuracy of the human evaluators' assessments and to maximize the ability to efficiently generate data for use in analyzing the accuracy of the human evaluators' assessments and the difficulty of the constructed response categories or questions. The systems and methods of the present invention utilize a storage means which relationally stores data regarding the human evaluators, the constructed responses, the scores awarded and the computer applications to utilize. The systems also comprise assessment stations and processing means for selectively transmitting constructed responses to the assessment stations and receiving scores awarded by the human evaluators assigned to the assessment stations.

Methods and systems for presentation and evaluation of constructed responses assessed by human evaluators

The present invention provide systems and methods for controlled presentation of constructed responses through a variety of computer applications to human evaluators for assessment. The systems and methods further provide for controlled presentation of the constructed responses to minimize the influences of psychometric factors on the accuracy of the human evaluators' assessments and to maximize the ability to efficiently generate data for use in analyzing the accuracy of the human evaluators' assessments and the difficulty of the constructed response categories or questions. The systems and methods of the present invention utilize a storage means which relationally stores data regarding the human evaluators, the constructed responses, the scores awarded and the computer applications to utilize. The systems also comprise assessment stations and processing means for selectively transmitting constructed responses to the assessment stations and receiving scores awarded by the human evaluators assigned to the assessment stations.

The present invention relates to computer-based technology in assessment particularly for the licensing and certification of professionals such as architects and engineers. For assessment of architects, the system includes tools and methods for item creation, computer programs for computerized item presentation, and programs for automatically scoring test responses by computer.

Computer-based simulation examination of architectural practice

Growing community of inventors

Hamilton, NJ, United States of America

Isaac I Bejar