Use of videotaping to evaluate pediatric resident performance of health supervision examinations of infants

Study objectives. To demonstrate reliability of a method for evaluating pediatric resident performance of health supervision examinations of infants using real patients and to evaluate pediatric resident performance of health supervision examinations of infants before and after an educational intervention consisting of a 6-month ambulatory rotation combined with video- assisted review of resident performance of the examination. Design. Longitudinal cohort study in which all second-year residents were enrolled. Residents preintervention (baseline) videotapes were compared with postintervention videotapes. Each resident served as his or her own control. Setting. University hospital pediatric resident continuity clinic. Subjects. Sixteen second-year pediatric residents who were participating in a required 6-month ambulatory rotation. Method and main results. Reliability study: Using a 51-item instrument derived from the American Academy of Pediatrics Guidelines for Health Supervision, trained raters blind to the sequence and dates of the videotaped health supervision examinations independently rated 44 resident encounters (a minimum of two raters per tape) and achieved good interrater reliability (κ = 0.80). Intervention study: After a 6-month ambulatory rotation which included resident-faculty reviews of preintervention videotapes, residents showed a 14% mean improvement in performance scores for the examination (P < .05). Conclusions. The results indicate that videotaped health supervision examinations using real patients can be reliably evaluated by observational techniques in a continuity practice. A 6-month ambulatory block rotation coupled with video-assisted interactive review of examinations is associated with measurable improvement in resident performance on the postintervention test.