• Version
• Download 122
• File Size 606.23 KB
• File Count 1
• Create Date August 2, 2018
• Last Updated August 2, 2018

Indices of Semantic Similarity for Automated Essay Scoring (AES)

Content is one of the main writing dimensions on which essays are judged and rated. Since no automated essay scoring (AES) system is capable (yet) of truly understanding the content of an essay and assessing its breadth, depth and relevance, AES systems use indirect methods and proxy indices for judging its quality. Most such indices are based on measures of semantic similarity between a given essay and some gold standard.The purpose of this study is to examine the efficiency (validity) of five computer-generated sematic indices used by NiteRater –, an AES system for text analysis and essay scoring of Hebrew texts (NiteRater, 2007). These indices can be classified into three categories: (1) indices based on semantic proximity between essays –, the similarity of an essay',s vocabulary to that of essays in various score-categories, (2) indices based on Principal Component Analysis (PCA) of semantic similarities, and (3) indices based on prompt-related vocabulary –, the similarity of the essay',s vocabulary to that of the prompt.Six essay-corpora of various genres were used to study the efficiency of the semantic indices, including essays written by native and non-native Hebrew speakers. The efficiency of these indices was assessed by correlating them with raters', scores. The internal structure of the semantic indices, as well as their differential validity for essays of different genres was also studied.The results of the study show that indices based on semantic proximity can capture a large proportion of the essay scores (r=.28-.85). These are followed by indices based on PCA of semantic similarities (r=.27-.57) and finally, by indices based on prompt-related vocabulary (r=-.51-.59), which are also the most sensitive to essay genre.

Attached Files