The relationship between the C-statistic and the accuracy of program-specific evaluations

The C-statistic of the risk-adjustment model is often used to judge the accuracy of program evaluations. However, the C-statistic depends on the variability in risk for individual transplants and may be inappropriate for determining the accuracy of program evaluations. A simulation study investigated the association of the C-statistic with several metrics of program evaluation accuracy, including categorizing programs into the 5-tier system and identifying programs for regulatory review. The simulation study used data from deceased donor kidney-alone transplants for adult recipients in the program-specific reports released January 2018. A range of C-statistics was generated by changing the variability in risk for individual transplants. The C-statistic had no association with any metric of program evaluation accuracy. Instead, the number of expected events at a program was the most important factor. For example, Spearman's rho, which is the correlation of ranks, was −0.27 and −0.72 between the true program-specific hazard ratios and assigned tiers for programs with, respectively, <3 and >10 expected events. Presence of unadjusted risk factors did not modify the associations, although the accuracy of program evaluations was systematically lower. Therefore, the C-statistic provides no information on the accuracy of program evaluations.