Educational Testing Service

Beata Beigman Klebanov

Michael Flor

Jill C Burstein

Chee Wee Leong

Nitin Madnani

John Sabatini

Michael Heilman

Joel Tetreault

Anastassia Loukina

Debanjan Ghosh

Xianyang Chen

Rene R Lawless

David Kaufer

Markku Tapio Hakkinen

Binod Gyawali

Paul L Yeoh

Daniel Blanchard

Adam Faulkner

Jennifer Lentini

Data is received that includes a passage of text. The passage of text can be tokenized so that features can be extracted from the resulting tokens. One or more machine learning models can detect one or more metaphors within the passage of text using the extracted features. The at least one machine learning model can be trained, for example, by interleaving data relating to metaphors with data relating to one or more auxiliary tasks associated with related figurative language constructs. Data can then be provided that identifies the detected metaphors. Related apparatus, systems, techniques and articles are also described.

Machine learning-based metaphor detection

Systems and methods are provided for automatic evaluation of argument critique essays written by young students in response to prompts. A transformer pre-trained for natural language processing is employed as a machine learning model, which is fine-tune with a first training dataset comprising unannotated argument critique essays written by college students, and then fine-tuned with a second training dataset comprising annotated argument critique essays written by middle school students, where each sentence in the second training dataset is annotated for the presence of valid critiques to prompts. The fine-tuned machine learning model is used to classify each sentence in an essay to be evaluated as either containing a valid critique or not.

Automatic evaluation of argumentative writing by young students

Reading comprehension of a user can be assessed by presenting, in a graphical user interface, sequential reading text comprising a plurality of passages. The graphical user interface can alternate between (i) automatically advancing through passages of the reading text and (ii) manually advancing through passages of the reading text within the graphical user interface which is in response to user-generated input received via the graphical user interface. An audio narration is provided during the automatic advancing of the reading text. An audio file is recorded during the manual advancing of the reading text which is used to automatically determine an estimated level of reading comprehension of the user. Data characterizing the determined level of reading comprehension of the user can then be provided (e.g., displayed, loaded into memory, stored on a hard drive, transmitted to a remote computing system, etc.). Related apparatus, systems, techniques and articles are also described.

E-book reading comprehension analytics

Systems and methods are provided for the detection of sentiment in writing. A plurality of texts is received from a larger collection of writing samples with a computer system. A set of seed words from the plurality of texts are labeled as being of positive sentiment or of negative sentiment with the computer system. The set of seed words is expanded in size with the computer system to provide an expanded set of seed words. Intensity values are assigned to words of the expanded set of seed words. Each of the words of the expanded set of seed words is assigned three intensity values: a value corresponding to the strength of the word's association with a positive polarity class, a value corresponding to the strength of the word's association with a negative polarity class, and a value corresponding to the strength of the word's association with a neutral polarity class.

Computer-implemented systems and methods for detection of sentiment in writing

Methods and systems for scoring an argument critique written essay, including identifying a discourse segment of the argument critique written essay, determining a position of each sentence in the discourse segment, classifying sentences into discourse units, evaluating an argumentative content of each sentence, and assigning an argumentative score to the essay based on the argumentative content of each sentence in the discourse segment of that essay. Methods and systems for training the scoring method are also disclosed. Corresponding apparatuses, systems, and methods are also disclosed.

Systems and methods for scoring argument critique written essays

Systems and methods of improving reading fluency by presenting a virtual reading partner that takes turns reading aloud with a reader, are presented. First audio data comprising speech corresponding to the reader reading aloud from a first portion of a written transcript is received. The first audio data is processed to identify spoken words from the speech. The reader's speaking of the words in the first portion of the written transcript are compared to correct speaking of words in the first portion of the written transcript. A reading fluency of the reader is assessed. Second audio data comprising a pre-recorded narration of a second portion of the written transcript or automatically generated text to speech of the second portion is generated. The virtual reading partner is presented to the reader and takes turns reading aloud with the reader until the end of the written transcript has been reached.

Virtual reading companion technology to support the acquisition of reading fluency

Computer-implemented systems and methods for generating a personalization score for a constructed response are provided. A constructed response generated by a user is received. The constructed response is processed with a processing system to generate one or more feature values representative of one or more aspects of the constructed response. A personalization score for the constructed response is generated using the processing system by applying a computer scoring model to the one or more feature values. The personalization score provides a measure of a personalization of the constructed response.

Systems and methods for generating a personalization score for a constructed response

Systems and methods are provided for determining whether an examination prompt elicits responses of a type for which the examination prompt is designed. Word types for each of a plurality of words in a plurality of responses to the examination prompt are identified. An evaluation distribution of the identified word types is determined. Predetermined distributions stored on a computer-readable medium associated with a plurality of different response types are accessed, and the evaluation distribution is compared to each of the predetermined distributions to determine which predetermined distribution the evaluation distribution is most similar. A determination is made as to whether the most similar distribution is associated with a same response type as the type for which the examination prompt is designed. An indication is made as to whether the examination prompt is deemed to elicit responses of the type for which the examination prompt is designed.

Systems and methods for determining the validity of an essay examination prompt

Methods and systems for scoring written text based on use of idiomatic expressions, including reading pre-selected idiomatic expressions in a canonical form into memory, expanding idiomatic expressions from the canonical form, reading a written response into the memory, pre-processing the written response, searching the pre-processed written response for idiomatic expressions, and assigning a score to the written response. The score may be based at least in part on the number of idiomatic expressions in the written response. Corresponding apparatuses, systems, and methods are also disclosed.

Systems and methods for automatic detection of idiomatic expressions in written responses

Systems and methods are provided for a computer-implemented method for identifying pairs of cohesive words within a text. A supervised model is trained to detect cohesive words within a text to be scored. Training the supervised model includes identifying a plurality of pairs of candidate cohesive words in a training essay and an order associated with the pairs of candidate cohesive words based on an order of words in the training essay. The pairs of candidate cohesive words are filtered to form a set of evaluation pairs. The evaluation pairs are provided via a graphical user interface based on the order associated with the pairs of candidate cohesive words. An indication of cohesion or no cohesion is received for the evaluation pairs via the graphical user interface. The supervised model is trained based on the evaluation pairs and the received indications.

Computer-implemented systems and methods for generating a supervised model for lexical cohesion detection

Computer-implemented systems and methods are provided for determining a document's complexity. For example, a computer performing the complexity analysis can receive a document. The computer can determine the content words within the document and determine an association measure for each group of content words. An association profile can be created for the document using the association measures. The computer can use the association profile to determine the complexity of the document. The complexity of the document may correspond to the document's suitable reading level or, if the document is an essay, an essay score.

Systems and methods for determining text complexity

Systems and methods described herein utilize supervised machine learning to generate a figure-of-speech prediction model for classify content words in a running text as either being figurative (e.g., as a metaphor, simile, etc.) or non-figurative (i.e., literal). The prediction model may extract and analyze any number of features in making its prediction, including a topic model feature, unigram feature, part-of-speech feature, concreteness feature, concreteness difference feature, literal context feature, non-literal context feature, and off-topic feature, each of which are described in detail herein. Since uses of figure of speech in writings may signal content sophistication, the figure-of-speech prediction model allows scoring engines to further take into consideration a text's use of figure of speech when generating a score.

Systems and methods for constructed response scoring using metaphor detection

Systems and methods are provided for identifying one or more target words of a corpus that have a lexical relationship to a plurality of provided cue words. The cue words and statistical lexical information derived from a corpus of documents are analyzed to determine candidate words that have a lexical association with the cue words. The statistical information includes numerical values indicative of probabilities of word pairs appearing together as adjacent words in a well-formed text or appearing together within a paragraph of a well-formed text. For each candidate word, a statistical association score between the candidate word and each of the cue words is determined. An aggregate score for each of the candidate words is determined based on the statistical association scores. One or more of the candidate words are selected to be the one or more target words based on the aggregate scores.

Growing community of inventors

Hopewell, NJ, United States of America

Beata Beigman Klebanov