• Version
• Download 178
• File Size 149.16 KB
• File Count 1
• Create Date August 2, 2018
• Last Updated August 2, 2018

Extended essay marking on screen: Is examiner marking accuracy influenced by marking mode?

Background: In the UK and elsewhere, large-scale educational assessment agencies are increasingly shifting towards having examiners mark examination scripts on screen rather than on paper. This shift has prompted questions about whether the mode of marking might influence examiner marking accuracy, particularly in relation to extended essay responses. This issue is important since it has implications for whether the stakeholders in large-scale assessments, including the candidates being assessed, can trust the marking outcomes if extended essays are marked on screen.Purpose: The study reported in this paper aimed to investigate empirically whether the mode in which extended essays are marked influences the accuracy of marking outcomes. This study was completed as part a wider research project which looked broadly at the influence of marking mode on examiners’, marking outcomes and processes for extended essays.Sample: A sample of 12 Advanced GCE examiners participated in the study. The examiners were all relatively experienced, holding between 6 and 31 total years’, experience (mean: 16.8 years) of marking for large-scale educational assessment agencies in the UK. Five of the examiners had some previous experience of marking essays on screen.Design and methods: One-hundred-and-eighty Advanced GCE American History ,examination essays were selected and split into two matched samples of 90 essays. The 180 essays were blind marked on paper by the examination’,s Principal Examiner (PE) to establish a study reference mark for each essay. Following training and standardisation, the sample of examiners each marked one 90-essay sample on paper and one 90-essay sample on screen. To control for essay sample and for marking order a crossover research design was used and the examiners were allocated to one of four examiner marking groups. Marking accuracy was defined as the extent of agreement between the examiner marks and the corresponding PE reference marks. Based on this definition, descriptive and general linear modelling statistical analyses were used to investigate whether the magnitude or direction of examiners’, marking accuracy was influenced by marking mode.Results: No association was found between marking mode and the magnitude of marking ,accuracy, but an extremely weak association was found between marking mode and thedirection of marking accuracy. This latter result was identified as both practically and statistically negligible. Overall the results presented no substantial evidence to indicate that marking accuracy for extended essays was influenced by marking mode.Conclusions: The results supported the conclusion that examiners are able to mark extended essays with equal accuracy on screen as they do on paper. The proposed practical implication of this conclusion is that extended essays can be marked on screen without compromising accuracy. The need for further investigation into the influence of ,marking mode on examiners’, marking processes for extended essays is highlighted.

Attached Files