• Version
• Download 155
• File Size 77.83 KB
• File Count 1
• Create Date August 2, 2018
• Last Updated August 2, 2018

No news is good news? Talking to the public about the reliability of assessment

Quantifying the reliability (or unreliability) of assessment results is a central part of the work of assessment professionals. Although much remains to be done, methods for calculating and reporting reliability indices have been widely discussed.Communicating with the public about unreliability in test scores has not been addressed to the same extent. In its initial public communications on this, Ofqual, the regulator of examinations and qualifications in England, has found that unreliability is a difficult topic for two reasons. Firstly, the concept of reliability is complex and hard to explain succinctly. Secondly, unreliability seems like an intrinsically bad news story.This paper will report on two sources of evidence. Firstly, literature describing the media environment that surrounds examination results in England will be summarised. Such literature can give a history of assessment organisations', attempts at communicating with the public, and make suggestions for how such bodies might communicate better. The second source of evidence is a series of workshops conducted to Ofqual',s specification by UK social research organisation Ipsos MORI. That work has given Ofqual an initial feel for the tolerance that different sectors of the public have for different sources of measurement inaccuracy in examination results.The paper will conclude by suggesting ways to improve each of the issues with unreliability as a media story, the problem of complexity will be addressed by allowing people to interact with the message via multiple media, using varied analogies and so on. In terms of the negativity of the story, the response will not be to try to make this into a good news story. Rather, the aspiration is to communicate the message that many assessment results contain an element of unreliability to the public in a manner that allows people to become more sophisticated users of those results.

Attached Files