TY - JOUR
T1 - Setting standard and defining quality of performance in the validation of a standardized-patient examination
AU - Rosebraugh, Curtis J.
AU - Speer, Alice J.
AU - Solomon, David J.
AU - Szauter, Karen E.
AU - Ainsworth, Michael A.
AU - Holden, Mark D.
AU - Lieberman, Steven A.
AU - Clyburn, Ernest B.
PY - 1997/11
Y1 - 1997/11
N2 - Purpose. To evaluate whether written standards increase the reproducibility of a physician-facilitated station in an objective structured clinical examination (OSCE) designed to assess history, physical-examination, and communication skills. Method. The OSCE examination at the University of Texas Medical Branch-Galveston consists of ten eight-minute stations. Six of these stations consist of three History, Physical-examination, Problem- solving, and Plan (HPPP) station pairs. Each existing clinical-problem HPPP station was given to two content experts to develop standards for faculty rating scales appropriate for the evaluation of third-year medical students. Three pairs of faculty members were used to determine interrater reliability by scoring videotapes of three HPPP stations' presentation and problem- solving components. Faculty pairs scored tapes of 15 students without using standards and tapes of 15 students using the standards developed. Differences between the reliabilities without and with the standards were tested for significance using Fisher's R to Z transformation. The reproducibility and standard error of measurement (SEM) were extrapolated for increasing amounts of testing time. The HPPP component scores were also correlated with the written examination scores and preceptors' ratings. Data were obtained from the three HPPP stations used in the 1995-96 internal medicine clerkship SP examination. Results. In all, 196 students completed the OSCE examination. The standards developed improved interrater reliability and reached statistical significance (p<.01) for one HPPP station. Reproducibility for the presentation and problem-solving components of HPPP stations were > .80 after five hours of testing. The problem-solving component correlated at .37 and .19 with written examinations and with ward grades, respectively. Conclusion. The data from this study suggest that standards increase the reproducibility of presentation and problem-solving components of an OSCE to a level as high as, or higher than, the associated with the history, physical-examination, and communication components of traditional standardized-patient examinations.
AB - Purpose. To evaluate whether written standards increase the reproducibility of a physician-facilitated station in an objective structured clinical examination (OSCE) designed to assess history, physical-examination, and communication skills. Method. The OSCE examination at the University of Texas Medical Branch-Galveston consists of ten eight-minute stations. Six of these stations consist of three History, Physical-examination, Problem- solving, and Plan (HPPP) station pairs. Each existing clinical-problem HPPP station was given to two content experts to develop standards for faculty rating scales appropriate for the evaluation of third-year medical students. Three pairs of faculty members were used to determine interrater reliability by scoring videotapes of three HPPP stations' presentation and problem- solving components. Faculty pairs scored tapes of 15 students without using standards and tapes of 15 students using the standards developed. Differences between the reliabilities without and with the standards were tested for significance using Fisher's R to Z transformation. The reproducibility and standard error of measurement (SEM) were extrapolated for increasing amounts of testing time. The HPPP component scores were also correlated with the written examination scores and preceptors' ratings. Data were obtained from the three HPPP stations used in the 1995-96 internal medicine clerkship SP examination. Results. In all, 196 students completed the OSCE examination. The standards developed improved interrater reliability and reached statistical significance (p<.01) for one HPPP station. Reproducibility for the presentation and problem-solving components of HPPP stations were > .80 after five hours of testing. The problem-solving component correlated at .37 and .19 with written examinations and with ward grades, respectively. Conclusion. The data from this study suggest that standards increase the reproducibility of presentation and problem-solving components of an OSCE to a level as high as, or higher than, the associated with the history, physical-examination, and communication components of traditional standardized-patient examinations.
UR - http://www.scopus.com/inward/record.url?scp=0030612885&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=0030612885&partnerID=8YFLogxK
U2 - 10.1097/00001888-199711000-00022
DO - 10.1097/00001888-199711000-00022
M3 - Article
C2 - 9387828
AN - SCOPUS:0030612885
SN - 1040-2446
VL - 72
SP - 1012
EP - 1014
JO - Academic Medicine
JF - Academic Medicine
IS - 11
ER -