Educational Testing Service

Jill C Burstein

Daniel Marcu

Martin S Chodorow

Chi Lu

Vyacheslav Andreyev

Derrick Higgins

Claudia Leacock

Susanne Wolff

Beata Beigman Klebanov

Swapna Somasundaran

Nitin Madnani

Yigal Attali

Lisa Braden-Harder

Slava Andreyev

Bruce A Kaplan

Donald A Rock

Karen Kukich

Claudia Gentile

Joel Tetreault

John Sabatini

Randy Mark Kaplan

Jane Shore

Adam Faulkner

Matthew Ventura

Systems and methods are provided for the detection of sentiment in writing. A plurality of texts is received from a larger collection of writing samples with a computer system. A set of seed words from the plurality of texts are labeled as being of positive sentiment or of negative sentiment with the computer system. The set of seed words is expanded in size with the computer system to provide an expanded set of seed words. Intensity values are assigned to words of the expanded set of seed words. Each of the words of the expanded set of seed words is assigned three intensity values: a value corresponding to the strength of the word's association with a positive polarity class, a value corresponding to the strength of the word's association with a negative polarity class, and a value corresponding to the strength of the word's association with a neutral polarity class.

Computer-implemented systems and methods for detection of sentiment in writing

Computer-implemented systems and methods for generating a personalization score for a constructed response are provided. A constructed response generated by a user is received. The constructed response is processed with a processing system to generate one or more feature values representative of one or more aspects of the constructed response. A personalization score for the constructed response is generated using the processing system by applying a computer scoring model to the one or more feature values. The personalization score provides a measure of a personalization of the constructed response.

Systems and methods for generating a personalization score for a constructed response

Systems and methods are provided for automatically scoring essay responses to a prompt using a scoring model. A relevant word corpus and an irrelevant word corpus are accessed. A scoring model is generated by, for each of a plurality of words in the relevant word corpus, determining a topic signature score based on a number of appearances of that word in the relevant word corpus and a number of appearances of that word in the irrelevant word corpus. For each of a plurality of words in an essay response, a topic signature score is determined for that word. A score for the essay response is determined based on the identified topic signature scores.

Systems and methods for determining the effectiveness of source material usage

Systems and methods are provided for automatically generating a coherence score for a text using a scoring model. A lexical chain is identified within a text to be scored, where the lexical chain comprises a set of words spaced within the text. A discourse element is identified within the text, where the discourse element comprises a word within the text. A coherence metric is determined based on a relationship between the lexical chain and the discourse element. A coherence score is generated using a scoring model by providing the coherence metric to the scoring model.

Computer-implemented systems and methods for measuring discourse coherence

In accordance with the teachings described herein, systems and methods are provided for measuring a user's comprehension of subject matter of a text. A summary generated by the user is received, where the summary summarizes the text. The summary is processed to determine a first numerical measure indicative of a similarity between the summary and a reference summary. The summary is processed to determine a second numerical measure indicative of a degree to which a single sentence of the summary summarizes an entirety of the text. The summary is processed to determine a third numerical measure indicative of a degree of copying in the summary of multi-word sequences present in the text. A numerical model is applied to the first numerical measure, the second numerical measure and the third numerical measure to determine a score for the summary indicative of the user's comprehension of the subject matter of the text.

System and method for automated scoring of a summary-writing task

To automatically annotate an essay, a sentence of the essay is identified and a feature associated with the sentence is determined. In addition, a probability of the sentence being a discourse element is determined by mapping the feature to a model. The model having been generated by a machine learning application based on at least one annotated essay. Furthermore, the essay is annotated based on the probability.

Automated annotation

Systems and methods for building a model for use in grading an essay are provided. A plurality of human graded essays are evaluated using a processor to generate a set of features. A score category is determined for each of the plurality of human graded essays. A weight is produced using a processor for each feature based on the score category for each of the plurality of human graded essays and the set of features. A model is generated using a processor based on the weights for the set of features for evaluating an essay. The set of features includes n features, where the first k features have optimized weights, and the last n&#x2212;k features have fixed predetermined weights.

Automatic essay scoring system

Systems and methods for building a trait model for essay evaluation are provided. At least one evaluated essay is received. A plurality of features pertaining to one or more traits from the at least one evaluated essay are identified and extracted. The one or more traits comprise writing errors, discourse, or vocabulary usage. A plurality of vector files based upon the plurality of features are created. A trait model for essay evaluation based upon the plurality of vector files is built. The trait model is evaluated, where the evaluation includes mapping features of a new essay to the trait model by navigating a multi-branched decision tree. At each branch of the decision tree, a value associated with the features of the new essay is used to determine how to proceed through the trait model.

Automated essay scoring

Systems and methods for creating a mathematical model for use in identifying discourse elements are described. A plurality of first essays relating to a particular subject are received, where each first essay is in an electronic format. Annotations for each first essay are received, where each annotation identifies at least one discourse element. Features are identified with a processor, where each feature is exhibited by at least one identified discourse element. Empirical frequencies are computed with a processor, where each empirical frequency relates to the presence of a feature with respect to the identified discourse elements across the plurality of first essays. Each empirical frequency is associated with the related identified discourse element with a processor. The empirical frequencies are utilized to select discourse elements in at least one second essay.

Methods for automated essay analysis

Systems and methods are provided for adapting a text, for example, for generating a text appropriate for the proficiency level of a language learner, are described. An original text is received and at least one portion of the original text to adapt is identified, where the adapted text will facilitate reader comprehension. An adaptation to be applied to the at least one portion of the original text is determined. The at least one portion of the original text is analyzed based on the adaptation, the adaptation is applied to the at least one portion of the original text, and the adapted text is provided.

Method and system for automatic generation of adapted content to facilitate reading skill development for language learners

A method and system for determining text coherence in an essay is disclosed. A method of evaluating the coherence of an essay includes receiving an essay having one or more discourse elements and text segments. The one or more discourse elements are annotated either manually or automatically. A text segment vector is generated for each text segment in a discourse element using sparse random indexing vectors. The method or system then identifies one or more essay dimensions and measures the semantic similarity of each text segment based on the essay dimensions. Finally, a coherence level is assigned to the essay based on the measured semantic similarities.

Method and system for determining text coherence

A method of grading an essay using an automated essay scoring system is provided. The method comprises the steps of deriving a set of predetermined features from the essay, wherein the predetermined feature set comprises one or more features that are independent from the test prompt, scoring the feature set with a scoring equation, wherein a multiple regression analysis with graded essay data produces weights for the scoring equation, generating a raw score for the essay; and processing the raw score for the essay into a score category based on an adaptive cutoff algorithm. Also provided is a method of generating a model in which to grade essays, wherein the data used to generate the model is independent from the test prompt or essay topic.

To automatically evaluate an essay, the essay is applied to a plurality of trait models and a plurality of trait scores are determined based on the plurality of trait models. Each of these trait scores having been generated from a respective trait model. In addition, a score is determined based on the plurality of trait scores.

An essay is analyzed automatically by accepting the essay and determining whether each of a predetermined set of features is present or absent in each sentence of the essay. For each sentence in the essay a probability that the sentence is a member of a certain discourse element category is calculated. The probability is based on the determinations of whether each feature in the set of features is present or absent. Furthermore, based on the calculated probabilities, a sentence is chosen as the choice for the discourse element category.

Growing community of inventors

Princeton, NJ, United States of America

Jill C Burstein