Setting standard and defining quality of performance in the validation of a standardized-patient examination

Curtis J. Rosebraugh, Alice J. Speer, David J. Solomon, Karen Szauter, Michael Ainsworth, Mark Holden, Steven Lieberman, Ernest B. Clyburn

Research output: Contribution to journalArticle

14 Citations (Scopus)

Abstract

Purpose. To evaluate whether written standards increase the reproducibility of a physician-facilitated station in an objective structured clinical examination (OSCE) designed to assess history, physical-examination, and communication skills. Method. The OSCE examination at the University of Texas Medical Branch-Galveston consists of ten eight-minute stations. Six of these stations consist of three History, Physical-examination, Problem- solving, and Plan (HPPP) station pairs. Each existing clinical-problem HPPP station was given to two content experts to develop standards for faculty rating scales appropriate for the evaluation of third-year medical students. Three pairs of faculty members were used to determine interrater reliability by scoring videotapes of three HPPP stations' presentation and problem- solving components. Faculty pairs scored tapes of 15 students without using standards and tapes of 15 students using the standards developed. Differences between the reliabilities without and with the standards were tested for significance using Fisher's R to Z transformation. The reproducibility and standard error of measurement (SEM) were extrapolated for increasing amounts of testing time. The HPPP component scores were also correlated with the written examination scores and preceptors' ratings. Data were obtained from the three HPPP stations used in the 1995-96 internal medicine clerkship SP examination. Results. In all, 196 students completed the OSCE examination. The standards developed improved interrater reliability and reached statistical significance (p<.01) for one HPPP station. Reproducibility for the presentation and problem-solving components of HPPP stations were > .80 after five hours of testing. The problem-solving component correlated at .37 and .19 with written examinations and with ward grades, respectively. Conclusion. The data from this study suggest that standards increase the reproducibility of presentation and problem-solving components of an OSCE to a level as high as, or higher than, the associated with the history, physical-examination, and communication components of traditional standardized-patient examinations.

Original languageEnglish (US)
Pages (from-to)1012-1014
Number of pages3
JournalAcademic Medicine
Volume72
Issue number11
StatePublished - Nov 1997

Fingerprint

Physical Examination
History
examination
performance
Students
history
Communication
Videotape Recording
Internal Medicine
Medical Students
Physicians
student
statistical significance
rating scale
communication skills
medical student
rating
physician
medicine
expert

ASJC Scopus subject areas

  • Public Health, Environmental and Occupational Health
  • Nursing(all)
  • Education

Cite this

Setting standard and defining quality of performance in the validation of a standardized-patient examination. / Rosebraugh, Curtis J.; Speer, Alice J.; Solomon, David J.; Szauter, Karen; Ainsworth, Michael; Holden, Mark; Lieberman, Steven; Clyburn, Ernest B.

In: Academic Medicine, Vol. 72, No. 11, 11.1997, p. 1012-1014.

Research output: Contribution to journalArticle

Rosebraugh, CJ, Speer, AJ, Solomon, DJ, Szauter, K, Ainsworth, M, Holden, M, Lieberman, S & Clyburn, EB 1997, 'Setting standard and defining quality of performance in the validation of a standardized-patient examination', Academic Medicine, vol. 72, no. 11, pp. 1012-1014.
Rosebraugh, Curtis J. ; Speer, Alice J. ; Solomon, David J. ; Szauter, Karen ; Ainsworth, Michael ; Holden, Mark ; Lieberman, Steven ; Clyburn, Ernest B. / Setting standard and defining quality of performance in the validation of a standardized-patient examination. In: Academic Medicine. 1997 ; Vol. 72, No. 11. pp. 1012-1014.
@article{5dbe3001c3c54d8c8574cc9c35da0bb6,
title = "Setting standard and defining quality of performance in the validation of a standardized-patient examination",
abstract = "Purpose. To evaluate whether written standards increase the reproducibility of a physician-facilitated station in an objective structured clinical examination (OSCE) designed to assess history, physical-examination, and communication skills. Method. The OSCE examination at the University of Texas Medical Branch-Galveston consists of ten eight-minute stations. Six of these stations consist of three History, Physical-examination, Problem- solving, and Plan (HPPP) station pairs. Each existing clinical-problem HPPP station was given to two content experts to develop standards for faculty rating scales appropriate for the evaluation of third-year medical students. Three pairs of faculty members were used to determine interrater reliability by scoring videotapes of three HPPP stations' presentation and problem- solving components. Faculty pairs scored tapes of 15 students without using standards and tapes of 15 students using the standards developed. Differences between the reliabilities without and with the standards were tested for significance using Fisher's R to Z transformation. The reproducibility and standard error of measurement (SEM) were extrapolated for increasing amounts of testing time. The HPPP component scores were also correlated with the written examination scores and preceptors' ratings. Data were obtained from the three HPPP stations used in the 1995-96 internal medicine clerkship SP examination. Results. In all, 196 students completed the OSCE examination. The standards developed improved interrater reliability and reached statistical significance (p<.01) for one HPPP station. Reproducibility for the presentation and problem-solving components of HPPP stations were > .80 after five hours of testing. The problem-solving component correlated at .37 and .19 with written examinations and with ward grades, respectively. Conclusion. The data from this study suggest that standards increase the reproducibility of presentation and problem-solving components of an OSCE to a level as high as, or higher than, the associated with the history, physical-examination, and communication components of traditional standardized-patient examinations.",
author = "Rosebraugh, {Curtis J.} and Speer, {Alice J.} and Solomon, {David J.} and Karen Szauter and Michael Ainsworth and Mark Holden and Steven Lieberman and Clyburn, {Ernest B.}",
year = "1997",
month = "11",
language = "English (US)",
volume = "72",
pages = "1012--1014",
journal = "Academic Medicine",
issn = "1040-2446",
publisher = "Lippincott Williams and Wilkins",
number = "11",

}

TY - JOUR

T1 - Setting standard and defining quality of performance in the validation of a standardized-patient examination

AU - Rosebraugh, Curtis J.

AU - Speer, Alice J.

AU - Solomon, David J.

AU - Szauter, Karen

AU - Ainsworth, Michael

AU - Holden, Mark

AU - Lieberman, Steven

AU - Clyburn, Ernest B.

PY - 1997/11

Y1 - 1997/11

N2 - Purpose. To evaluate whether written standards increase the reproducibility of a physician-facilitated station in an objective structured clinical examination (OSCE) designed to assess history, physical-examination, and communication skills. Method. The OSCE examination at the University of Texas Medical Branch-Galveston consists of ten eight-minute stations. Six of these stations consist of three History, Physical-examination, Problem- solving, and Plan (HPPP) station pairs. Each existing clinical-problem HPPP station was given to two content experts to develop standards for faculty rating scales appropriate for the evaluation of third-year medical students. Three pairs of faculty members were used to determine interrater reliability by scoring videotapes of three HPPP stations' presentation and problem- solving components. Faculty pairs scored tapes of 15 students without using standards and tapes of 15 students using the standards developed. Differences between the reliabilities without and with the standards were tested for significance using Fisher's R to Z transformation. The reproducibility and standard error of measurement (SEM) were extrapolated for increasing amounts of testing time. The HPPP component scores were also correlated with the written examination scores and preceptors' ratings. Data were obtained from the three HPPP stations used in the 1995-96 internal medicine clerkship SP examination. Results. In all, 196 students completed the OSCE examination. The standards developed improved interrater reliability and reached statistical significance (p<.01) for one HPPP station. Reproducibility for the presentation and problem-solving components of HPPP stations were > .80 after five hours of testing. The problem-solving component correlated at .37 and .19 with written examinations and with ward grades, respectively. Conclusion. The data from this study suggest that standards increase the reproducibility of presentation and problem-solving components of an OSCE to a level as high as, or higher than, the associated with the history, physical-examination, and communication components of traditional standardized-patient examinations.

AB - Purpose. To evaluate whether written standards increase the reproducibility of a physician-facilitated station in an objective structured clinical examination (OSCE) designed to assess history, physical-examination, and communication skills. Method. The OSCE examination at the University of Texas Medical Branch-Galveston consists of ten eight-minute stations. Six of these stations consist of three History, Physical-examination, Problem- solving, and Plan (HPPP) station pairs. Each existing clinical-problem HPPP station was given to two content experts to develop standards for faculty rating scales appropriate for the evaluation of third-year medical students. Three pairs of faculty members were used to determine interrater reliability by scoring videotapes of three HPPP stations' presentation and problem- solving components. Faculty pairs scored tapes of 15 students without using standards and tapes of 15 students using the standards developed. Differences between the reliabilities without and with the standards were tested for significance using Fisher's R to Z transformation. The reproducibility and standard error of measurement (SEM) were extrapolated for increasing amounts of testing time. The HPPP component scores were also correlated with the written examination scores and preceptors' ratings. Data were obtained from the three HPPP stations used in the 1995-96 internal medicine clerkship SP examination. Results. In all, 196 students completed the OSCE examination. The standards developed improved interrater reliability and reached statistical significance (p<.01) for one HPPP station. Reproducibility for the presentation and problem-solving components of HPPP stations were > .80 after five hours of testing. The problem-solving component correlated at .37 and .19 with written examinations and with ward grades, respectively. Conclusion. The data from this study suggest that standards increase the reproducibility of presentation and problem-solving components of an OSCE to a level as high as, or higher than, the associated with the history, physical-examination, and communication components of traditional standardized-patient examinations.

UR - http://www.scopus.com/inward/record.url?scp=0030612885&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0030612885&partnerID=8YFLogxK

M3 - Article

C2 - 9387828

AN - SCOPUS:0030612885

VL - 72

SP - 1012

EP - 1014

JO - Academic Medicine

JF - Academic Medicine

SN - 1040-2446

IS - 11

ER -