Educational Testing Service

Yigal Attali

Jill C Burstein

Vyacheslav Andreyev

Michael Flor

Paul Douglas Deane

Mo Zhang

Gary Feng

Jiangang Hao

Yoav Bergner

Michael E Wagner

Nathan Lederer

Systems and methods for training raters to rate constructed responses to tasks are described herein. In one embodiment, a plurality of trainee raters are selected without regard to their prior experience. The trainee raters are then train in individual training sessions, during which they are asked to rate responses to a task. Each session presents to the trainee rater the task, a rating rubric, and training responses to the task. The training program receives ratings assigned by the trainee rater to the training responses through a graphical user interface. Upon receiving the assigned rating, the training program presents feedback substantially immediately and determines a score for the trainee rater's assigned rating. Thereafter, qualified raters are selected from the plurality of trainee raters based upon their performance during the training sessions as compared with a statistical model. Operational constructed responses are then assigned to rated by the qualified raters.

Systems and methods for computer-based training of crowd-sourced raters

Systems and methods are provided for scoring a constructed response generated by a user and providing information on the user's writing behavior. A constructed response and associated electronic process log are received. The constructed response is processed to generate first feature values representative of aspects of the constructed response. The electronic process log is processed to generate second feature values related to the user's actions in generating the constructed response. A score for the constructed response is generated using the processing system by applying a computer scoring model to the first and second feature values. A rule of a rule engine that is satisfied is identified, the rule being satisfied when one or more feature values of the second feature values meet a condition associated with the rule. Information on the user's actions in generating the constructed response is provided based on the satisfied rule.

Generating scores and feedback for writing assessment and instruction using electronic process logs

A method and system for customizing an automated essay scoring system are disclosed. Relative weights may be assigned to a plurality of features for a plurality of benchmark essay responses. Based on the relative weights, automated scores may be determined for one or more first essay responses to a first prompt. The automated scores may be scaled based on one or more scoring standards for the first prompt. A scaled automated score may be assigned for a second essay response to the first prompt based on the scaled automated scores for the one or more first essay responses.

Method of model scaling for an automated essay scoring system

Systems and methods for building a model for use in grading an essay are provided. A plurality of human graded essays are evaluated using a processor to generate a set of features. A score category is determined for each of the plurality of human graded essays. A weight is produced using a processor for each feature based on the score category for each of the plurality of human graded essays and the set of features. A model is generated using a processor based on the weights for the set of features for evaluating an essay. The set of features includes n features, where the first k features have optimized weights, and the last n&#x2212;k features have fixed predetermined weights.

Automatic essay scoring system

A method, system and processor-readable storage medium for evaluating vocabulary similarity are disclosed. A generic rate may be determined for each word in a plurality of first responses. Each first response may respond to one of a plurality of first prompts. At least one first response may respond to each of the first prompts. A specific rate may be determined for each word in a plurality of second responses, which each respond to a second prompt. A target response may be received that is associated with the second prompt and has a plurality of words. A vocabulary similarity index may be computed for the target response based on one or more generic rates and on or more specific rates. A determination of whether the target response is off-topic may be made based on the vocabulary similarity index for the target response.

Method and system for evaluating vocabulary similarity

A method of grading an essay using an automated essay scoring system is provided. The method comprises the steps of deriving a set of predetermined features from the essay, wherein the predetermined feature set comprises one or more features that are independent from the test prompt, scoring the feature set with a scoring equation, wherein a multiple regression analysis with graded essay data produces weights for the scoring equation, generating a raw score for the essay; and processing the raw score for the essay into a score category based on an adaptive cutoff algorithm. Also provided is a method of generating a model in which to grade essays, wherein the data used to generate the model is independent from the test prompt or essay topic.

Growing community of inventors

Lawrenceville, NJ, United States of America

Yigal Attali