• Version
• Download 125
• File Size 384.77 KB
• File Count 1
• Create Date August 2, 2018
• Last Updated August 2, 2018

Comparing the reliability of standard maintaining via examiner judgement to statistical approaches

This paper compares the reliability of two methods for maintaining examination standards - a qualitative comparison of scripts in different years by expert judges, and the use of statistical predictions. Initially, the paper will introduce a theoretical model for the way in which expert judgement operates. This model encompasses the relationship between the score awarded to an examination script and the perceived holistic quality of the script by an expert judge, as well as the expected level of inter-judge variation in the level of perceived holistic quality for a single script. The paper will then demonstrate that this model provides results that are consistent with existing research, including both studies that are critical and studies that are supportive of the use of expert judgement. The model will then be applied to examine how the reliability of expert-driven approaches to standard maintaining is dependent upon the number of judges involved and the number of candidate scripts that are scrutinised. Finally the expected reliabilities of grade boundaries derived via differing approaches to expert judgement will be compared to a purely statistical approach to the same problem.Keywords: Standard maintaining, comparative judgement, reliability

Attached Files