Educational Testing Service

Michael Heilman

Derrick Higgins

Nitin Madnani

Klaus Zechner

Lei Chen

Michael Flor

Martin S Chodorow

Beata Beigman Klebanov

Isaac I Bejar

Chee Wee Leong

Yoko Futagi

Joel Tetreault

F Jay Breyer

Systems and methods are provided for automatically scoring a constructed response. The constructed response is processed to generate a plurality of numerical vectors that is representative of the constructed response. A model is applied to the plurality of numerical vectors. The model includes an input layer configured to receive the plurality of numerical vectors, the input layer being connected to a following layer of the model via a first plurality of connections. Each of the connections has a first weight. An intermediate layer of nodes is configured to receive inputs from an immediately-preceding layer of the model via a second plurality of connections, each of the connections having a second weight. An output layer is connected to the intermediate layer via a third plurality of connections, each of the connections having a third weight. The output layer is configured to generate a score for the constructed response.

Deep convolutional neural networks for automated scoring of constructed responses

A computer-implemented method of training an assessment model for assessing constructed texts expressing opinions on subjects includes accessing a plurality of training texts, which are constructed texts. The training texts are analyzed with the processing system to derive values of a plurality of linguistic features of an assessment model. At least one of the plurality of linguistic features relates to sentiment and at least one of the plurality of linguistic feature relates to specificity. The assessment model is trained with the processing system based on the values of the plurality of linguistic features. Based on the training, a weight for each of the plurality of linguistic features is determined. The assessment model is calibrated to include the weights for at least some of the plurality of linguistic features such that the assessment model is configured to generate assessment measures for constructed texts expressing opinions on subjects.

Systems and methods for automatically assessing constructed recommendations based on sentiment and specificity measures

Systems and methods are provided for determining a susceptibility of a computer-implemented automated scoring engine to gaming strategies. A plurality of responses to a prompt are provided to a computer-implemented automated scoring engine to receive a first set of scores. A first transformation is performed on each of the plurality of responses to generate a first set of transformed responses. The first set of transformed responses is provided to the computer-implemented automatic scoring engine to receive a second set of scores, and a gaming susceptibility metric is determined based on the first set of scores and the second set of scores.

Systems and methods for evaluating susceptibility of automated scoring engines to gaming behavior

Systems and methods described herein utilize supervised machine learning to generate a figure-of-speech prediction model for classify content words in a running text as either being figurative (e.g., as a metaphor, simile, etc.) or non-figurative (i.e., literal). The prediction model may extract and analyze any number of features in making its prediction, including a topic model feature, unigram feature, part-of-speech feature, concreteness feature, concreteness difference feature, literal context feature, non-literal context feature, and off-topic feature, each of which are described in detail herein. Since uses of figure of speech in writings may signal content sophistication, the figure-of-speech prediction model allows scoring engines to further take into consideration a text's use of figure of speech when generating a score.

Systems and methods for constructed response scoring using metaphor detection

In accordance with the teachings described herein, systems and methods are provided for identifying organizational elements in argumentative or persuasive discourse. A text that has been annotated is received. The annotated text includes argumentative or persuasive discourse that includes claims and evidence and organizational elements configured to organize the claims and evidence. Annotations of the annotated text distinguish the organizational elements from the claims and evidence. A rule set or a feature set is identified from the annotated text, where the rule set or the feature set includes textual patterns or word frequency features related to the organizational elements of the annotated text. A model is built based on the annotations and on the rule set or the feature set. The model is configured to identify organizational elements in a new text. The model is applied to the new text.

Growing community of inventors

Princeton, NJ, United States of America

Michael Heilman