Cut-scores revisited

Feasibility of a new method for group standard setting

Boaz Shulruf, Lee Coombes, Arvin Damodaran, Adrian Freeman, Philip Jones, Steven Lieberman, Phillippa Poole, Joel Rhee, Tim Wilkinson, Peter Harris

Research output: Contribution to journalArticle

Abstract

Background: Standard setting is one of the most contentious topics in educational measurement. Commonly-used methods all have well reported limitations. To date, there is not conclusive evidence suggesting which standard setting method yields the highest validity. Methods: The method described and piloted in this study asked expert judges to estimate the scores on a real MCQ examination that they consider indicated a clear pass, clear fail, and pass mark for the examination as a whole. The mean and SD of the judges responses to these estimates, Z scores and confidence intervals were used to derive the cut-score and the confidence in it. Results: In this example the new method's cut-score was higher than the judges' estimate. The method also yielded estimates of statistical error which determine the range of the acceptable cut-score and the estimated level of confidence one may have in the accuracy of that cut-score. Conclusions: This new standard-setting method offers some advances, and possibly advantages, in that the decisions being asked of judges are based on firmer constructs, and it takes into account variation among judges.

Original languageEnglish (US)
Article number126
JournalBMC Medical Education
Volume18
Issue number1
DOIs
StatePublished - Jun 7 2018
Externally publishedYes

Fingerprint

Group
confidence
examination
expert
evidence

Keywords

  • Angoff
  • Assessment
  • MCQ
  • Standard setting

ASJC Scopus subject areas

  • Education

Cite this

Shulruf, B., Coombes, L., Damodaran, A., Freeman, A., Jones, P., Lieberman, S., ... Harris, P. (2018). Cut-scores revisited: Feasibility of a new method for group standard setting. BMC Medical Education, 18(1), [126]. https://doi.org/10.1186/s12909-018-1238-7

Cut-scores revisited : Feasibility of a new method for group standard setting. / Shulruf, Boaz; Coombes, Lee; Damodaran, Arvin; Freeman, Adrian; Jones, Philip; Lieberman, Steven; Poole, Phillippa; Rhee, Joel; Wilkinson, Tim; Harris, Peter.

In: BMC Medical Education, Vol. 18, No. 1, 126, 07.06.2018.

Research output: Contribution to journalArticle

Shulruf, B, Coombes, L, Damodaran, A, Freeman, A, Jones, P, Lieberman, S, Poole, P, Rhee, J, Wilkinson, T & Harris, P 2018, 'Cut-scores revisited: Feasibility of a new method for group standard setting', BMC Medical Education, vol. 18, no. 1, 126. https://doi.org/10.1186/s12909-018-1238-7
Shulruf, Boaz ; Coombes, Lee ; Damodaran, Arvin ; Freeman, Adrian ; Jones, Philip ; Lieberman, Steven ; Poole, Phillippa ; Rhee, Joel ; Wilkinson, Tim ; Harris, Peter. / Cut-scores revisited : Feasibility of a new method for group standard setting. In: BMC Medical Education. 2018 ; Vol. 18, No. 1.
@article{1e14a70f3e314bfb9ec4ee5f7bf58fba,
title = "Cut-scores revisited: Feasibility of a new method for group standard setting",
abstract = "Background: Standard setting is one of the most contentious topics in educational measurement. Commonly-used methods all have well reported limitations. To date, there is not conclusive evidence suggesting which standard setting method yields the highest validity. Methods: The method described and piloted in this study asked expert judges to estimate the scores on a real MCQ examination that they consider indicated a clear pass, clear fail, and pass mark for the examination as a whole. The mean and SD of the judges responses to these estimates, Z scores and confidence intervals were used to derive the cut-score and the confidence in it. Results: In this example the new method's cut-score was higher than the judges' estimate. The method also yielded estimates of statistical error which determine the range of the acceptable cut-score and the estimated level of confidence one may have in the accuracy of that cut-score. Conclusions: This new standard-setting method offers some advances, and possibly advantages, in that the decisions being asked of judges are based on firmer constructs, and it takes into account variation among judges.",
keywords = "Angoff, Assessment, MCQ, Standard setting",
author = "Boaz Shulruf and Lee Coombes and Arvin Damodaran and Adrian Freeman and Philip Jones and Steven Lieberman and Phillippa Poole and Joel Rhee and Tim Wilkinson and Peter Harris",
year = "2018",
month = "6",
day = "7",
doi = "10.1186/s12909-018-1238-7",
language = "English (US)",
volume = "18",
journal = "BMC Medical Education",
issn = "1472-6920",
publisher = "BioMed Central",
number = "1",

}

TY - JOUR

T1 - Cut-scores revisited

T2 - Feasibility of a new method for group standard setting

AU - Shulruf, Boaz

AU - Coombes, Lee

AU - Damodaran, Arvin

AU - Freeman, Adrian

AU - Jones, Philip

AU - Lieberman, Steven

AU - Poole, Phillippa

AU - Rhee, Joel

AU - Wilkinson, Tim

AU - Harris, Peter

PY - 2018/6/7

Y1 - 2018/6/7

N2 - Background: Standard setting is one of the most contentious topics in educational measurement. Commonly-used methods all have well reported limitations. To date, there is not conclusive evidence suggesting which standard setting method yields the highest validity. Methods: The method described and piloted in this study asked expert judges to estimate the scores on a real MCQ examination that they consider indicated a clear pass, clear fail, and pass mark for the examination as a whole. The mean and SD of the judges responses to these estimates, Z scores and confidence intervals were used to derive the cut-score and the confidence in it. Results: In this example the new method's cut-score was higher than the judges' estimate. The method also yielded estimates of statistical error which determine the range of the acceptable cut-score and the estimated level of confidence one may have in the accuracy of that cut-score. Conclusions: This new standard-setting method offers some advances, and possibly advantages, in that the decisions being asked of judges are based on firmer constructs, and it takes into account variation among judges.

AB - Background: Standard setting is one of the most contentious topics in educational measurement. Commonly-used methods all have well reported limitations. To date, there is not conclusive evidence suggesting which standard setting method yields the highest validity. Methods: The method described and piloted in this study asked expert judges to estimate the scores on a real MCQ examination that they consider indicated a clear pass, clear fail, and pass mark for the examination as a whole. The mean and SD of the judges responses to these estimates, Z scores and confidence intervals were used to derive the cut-score and the confidence in it. Results: In this example the new method's cut-score was higher than the judges' estimate. The method also yielded estimates of statistical error which determine the range of the acceptable cut-score and the estimated level of confidence one may have in the accuracy of that cut-score. Conclusions: This new standard-setting method offers some advances, and possibly advantages, in that the decisions being asked of judges are based on firmer constructs, and it takes into account variation among judges.

KW - Angoff

KW - Assessment

KW - MCQ

KW - Standard setting

UR - http://www.scopus.com/inward/record.url?scp=85048192807&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85048192807&partnerID=8YFLogxK

U2 - 10.1186/s12909-018-1238-7

DO - 10.1186/s12909-018-1238-7

M3 - Article

VL - 18

JO - BMC Medical Education

JF - BMC Medical Education

SN - 1472-6920

IS - 1

M1 - 126

ER -