Interrater reliability of therapists' judgments of graphed data

Increased emphasis on the use of single-subject designs in physical therapy research suggests the need to examine whether therapists can meaningfully interpret the results of such research as part of the clinical decision-making process. With this goal in mind, the interrater reliability of therapists to make visual judgments from graphed data that included a trend line was examined. Thirty therapists were presented with 24 graphs of single-subject data from AB (baseline-treatment) designs. Each graph included a trend line calculated using the split-middle method of trend estimation. The trend line was computed using the baseline data and then extended into the treatment phase to 'predict' patient performance. The analysis, using intraclass correlation coefficients (ICCs), revealed low interrater agreement, with ICC values ranging from .37 to .55 for the entire sample. Evidence is presented that the statistical backgrounds of some raters positively influenced interrater reliability. No statistically significant relationship was found between interrater agreement and visual components of the graphed data, such as changes in slope or variability.