Educational Testing Service

Paul Douglas Deane

Mo Zhang

Jiangang Hao

Chen Li

Martin S Chodorow

Michael Flor

Derrick Higgins

Yoko Futagi

Yigal Attali

Gary Feng

Brent Bridgeman

Randy E Bennett

Chaitanya Ramineni

Ick Kyu Choi

Amanda Roth

Yoav Bergner

Hongwen Guo

Peter Van Rijn

Michael E Wagner

Eowyn Winchester

Nathan Lederer

Theresa Richter

Robert Krovetz

A method comprising accessing a first data structure that is associated with a first product prepared by a student and that includes first process data associated with a process performed by the student in generating the first product, analyzing the first data structure to generate a first characterization score based on the first product and the first process data, accessing a second data structure that is associated with a second product prepared by the student and that includes second process data associated with a process performed by the student in generating the second product, analyzing the second data structure to generate a second characterization score based on the second product and the second process data, and calculating a skill level change metric based on the first characterization score and the second characterization score indicating a change in ability level of the student over a course of the scenario-based assessment.

Platform for characterizing performance on a scenario-based assessment to school differences in curriculum

Biometric keystroke measure data derived from a computer-implemented long form examination taken by an examinee is received. Features are the extracted from the biometric keystroke measure data for the examinee. A similarity value is then determined, using one or more of a direct distance approach or a machine learning approach, for the extracted features relative to features extracted from biometric keystroke measure data derived from each of a plurality of other examinees while taking the long form examination. At least one of the determined similarity values is then identified having a value above a pre-defined threshold. The pre-defined threshold indicates a likelihood of the examinee being the same as one of the other examinees. Data can then be provided that characterizes the identification. Related apparatus, systems, techniques and articles are also described.

Behavior-based electronic essay assessment fraud detection

Systems and methods for defining and using an optimal burst boundary threshold to assess the reliability of a manual/automatic writing score are presented. Keystroke data, including inter-key interval data, such as inter-word interval data, may be gathered from writings. Clustering analyses may be performed on the inter-key interval data to determine an optimal number of bursts for the writings. An optimal burst boundary may be determined from the optimal number of bursts. Other burst-related measures and statistics, including the average and maximum burst lengths, may be determined from the writings based on the optimal burst boundary threshold. A score may be received for each of the writings. A validation indication metric may be generated for each of the writings based on the received score and the optimal burst boundary threshold. The resulting measures and statistics may be used or applied in different ways and provide personalized feedback as learning analytics.

Defining personalized writing burst measures of translation using keystroke logs

Systems and methods are provided for providing feedback on a user's writing behavior in generating a constructed response. An electronic process log for the constructed response is received. The electronic process log is processed to generate a vector having a predetermined number of elements. The vector comprises information related to the user's actions in generating the constructed response and includes (i) data indicating types of actions performed by the user in generating the constructed response, (ii) time points associated with the actions, and (iii) locations associated with the actions, each location indicating whether an associated action occurred within a word, between words, within a phrase, between phrases, within a sentence, between sentences, within a paragraph, or between paragraph. The vector is compared to one or more vectors associated with other constructed responses, and feedback is generated based on the comparison.

Generating scores and feedback for writing assessment and instruction using electronic process logs

A computer-implemented method of calibrating an assessment model for assessing responses includes accessing a plurality of training responses with a processing system for training an assessment model. The processing system analyzes the plurality of training responses to derive values of multiple features of the training responses. The processing system trains the assessment model based on the values of the multiple features of the training responses and a portfolio score for each individual associated with the plurality of training responses utilized in the training. The portfolio score for each individual corresponds to a measure of proficiency based on multiple writing samples constructed by the individual. The processing system determines, based on said training, a weight for each of the multiple features. The processing system calibrates the assessment model to include the weights for at least some of the features such that the assessment model is configured to generate scores for responses.

Systems and methods for generating automated evaluation models

A method and system for using a natural language generator for automatic assessment item generation is disclosed. The natural language generator includes a document structure generator that produces an abstract document specification defining a structure for an assessment item based on user input. The abstract document specification is input into a logical schema generator, which produces a logical schema specification that creates a more detailed specification for an assessment item. Finally, a sentence generator receives the logical schema specification and creates natural language for the assessment item based on the variables defined in the logical schema specification.

Automated test item generation system and method

Systems and methods are provided for scoring a constructed response generated by a user and providing information on the user's writing behavior. A constructed response and associated electronic process log are received. The constructed response is processed to generate first feature values representative of aspects of the constructed response. The electronic process log is processed to generate second feature values related to the user's actions in generating the constructed response. A score for the constructed response is generated using the processing system by applying a computer scoring model to the first and second feature values. A rule of a rule engine that is satisfied is identified, the rule being satisfied when one or more feature values of the second feature values meet a condition associated with the rule. Information on the user's actions in generating the constructed response is provided based on the satisfied rule.

Systems and methods are provided for non-monotonic recognition of phrasal terms. Phrasal terms are identified from a corpus of written materials and ranked based on, for example, a mutual rank ratio. The phrasal terms are sequentially selected and a determination is made as to whether to accept or reject the selected phrasal term based on at least one predetermined criteria. The ranking of the phrasal terms may also rely on linguistic support to reduce duplication of phrasal terms and to distinguish different confidence levels for identified and accepted phrasal terms.

Computer-implemented systems and methods for non-monotonic recognition of phrasal terms

Systems and methods for detecting collocation errors in a text sample using a reference database from a corpus are provided. Collocation candidates are identified within the text sample based upon syntactic patterns in the text sample. Whether a given collocation candidate contains a collocation error is detected, the detecting including: determining a first association measure using the reference database for the given collocation candidate; determining whether the first association measure satisfies a predetermined condition and identifying the given collocation candidate as proper if the first association measure satisfies the predetermined condition; determining an additional association measure for a variation of the given collocation candidate using the reference database; and determining whether or not the collocation candidate contains an error based upon the additional association measure of the variation.

Systems and methods for identifying collocation errors in text

A method and system for performing automatic text analysis is described. A local ranking for one or more contexts with respect to a word and a global ranking for one or more contexts are generated. The rankings are based on the frequency with which the contexts appear in a corpus. A statistic may be generated using the local and global rankings, such as a log ratio rank statistic equal to the logarithm of the global rank divided by local rank, to measure the similarity of contexts with respect to words with which they combine. A source matrix of word to context values is then created. Singular value decomposition is used to create sub-matrices from the source matrix. Vectors from the sub-matrices corresponding to context(s) and/or word(s) are then selected to determine term-term or context-context similarity or term-context correspondence.

Cooccurrence and constructions

A method and system for determining a lexical association of phrasal terms are described. A corpus having a plurality of words is received, and a plurality of contexts including one or more context words proximate to a word in the corpus is determined. An occurrence count for each context is determined, and a global rank is assigned based on the occurrence count. Similarly, a number of occurrences of a word being used in a context is determined, and a local rank is assigned to the word-context pair based on the number of occurrences. A rank ratio is then determined for each word-context pair. A rank ratio is equal to the global rank divided by the local rank for a word-context pair. A mutual rank ratio is determined by multiplying the rank ratios corresponding to a phrase. The mutual rank ratio is used to identify phrasal terms in the corpus.

Growing community of inventors

Lawrence Township, NJ, United States of America

Paul Douglas Deane