# How to pool c-statistic/AUROC (or any bounded variable) after using multiple imputation techniques?

I am conducting a study where I am interested in predicting a dichotomous outcome (poor outcome yes/no) for patients in a hospital setting. Specifically, I want to compare how different summary measures for the first week of admission affect the models' discrimination, as measured by the c-index (aka area under the receiver operator curve or AUROC).

As usually happens in clinical studies however, I have missing data on predictor and outcome variables. I have decided to attack this problem by using multiple imputation techniques. This way I have created 50 datasets with replaced missing values (using the 'mice' package in R).

Using the appropriate functions I am able to obtain the c-statistics with confidence interval (& variance) for each imputation dataset. Using 'plain' Rubins rules for pooling of normally distributed variables I would now average the point estimate and adjust the total variance for the variance between imputation datasets. Now I come onto the problem: I am unsure whether I can treat the 50 c-indices as normally distributed and calculate point estimate and the variance needed for a proper confidence interval.

I have tried searching for an answer, but I only found the following three suggestions used in (slightly) different situations:

1. to pool assuming normal distribution anyway (often applied to other statistics which are bounded or definitively not normally distributed);
2. look at the distribution of statistics over all imputation datasets and take the median c-index as point estimate, while using the 2.5th and 97.5th percentile values as lower and upper bound of a 95% confidence interval.
3. transform all c-indices and variances to an unbounded scale, pool transformed values assuming normal distribution, and finally transform back to bounded c-index scale (as suggested for the observed:expected ratio by log-transforming in Siregar S - Eur J Cardiothorac Surg 2012). For the $[0, 1]$ bounded c-index this could be done by logit-transformation of the c-indices.

Any help would be greatly appreciated.

• I would favor option 1, as it is often done in diagnostic meta-analyses pooling AUCs of ROCs, as long as your point estimates are reasonably distant from 1(eg ncbi.nlm.nih.gov/pmc/articles/PMC4644739). – Joe_74 Feb 3 '17 at 12:09
• @Joe_74 thanks for your comment. Option one would be the easiest too. However, If I understand correctly, the meta-analysis reference you are referring to uses modelling of the distributions of sensitivity&specificity found across the included studied in order to build a 'HSROCurve' which combines and estimates the general diagnostic performance underlying these studies, instead of pooling c-index or other performance measures directly. Could you maybe point out where the authors refer to something alike pooling according to option 1? – IWS Feb 3 '17 at 15:10
• I think that in that article they say you can pool sensitivity/specificity separately, and accordingly AUC of ROC. In a meta-analysis context, you could for instance use a generic inverse variance weighting method, as long as you have point estimates remote from 1 and small standard errors. Thus a similar approach to Rubin's rule in your case is reasonable. – Joe_74 Feb 3 '17 at 15:24

The c-index is a useful measure of predictive discrimination because it is easy to interpret and at least moderately sensitive. It is not a full-information proper accuracy scoring rule. It is not sensitive enough for comparing two models. So I suggest you obtain the best model using all the partial information available (e.g., multiple imputation with the number of imputations being at least the percentage of records that are incomplete), then attempt to quantify the value of that single model. That is easier said than done, but you can start with the overall Wald statistic for the global null hypothesis that none of the predictors are associated with $Y$. There are a few papers showing how to derive a unitless discrimination index from the Wald $\chi^2$ statistic. Also take a quick look at the $g$-index in my Regression Modeling Strategies book and notes.

• Just curious, but why do you prefer the global Wald test to the global likelihood ratio test? My default would have been the latter. – gung - Reinstate Monica Jan 17 '17 at 20:14
• Thanks for posing the question related to the blog @gung. I would always prefer the likelihood ratio test but don't have experience in using its more complicated version when multiple imputation is involved. Great point. – Frank Harrell Jan 17 '17 at 21:15
• @FrankHarrell Altough I am very inclined to follow your advice (and accept this as an answer), I must point out that this answer does not in fact answer my question. Unless you would advocate never to use the c-statistic, I am still curious whether 'regular' Rubins rules apply to bounded values, or whether there is any other way of pooling them. Would you elaborate on this? – IWS Jan 18 '17 at 7:51

After asking and looking around, I've been pointed at the following reference concerning the meta-analyses of prediction models (in biomedical research) by Debray TPA et al in the British Medical Journal 2016.

In appendix 9 the authors provide an explanation of how to pool multiple c-indices across different studies and specify how to obtain the total variance and its components (within and between variance). All of this is based on transforming the values using a logit transformation as a first step. Second, pooling across studies occurs in a similar way as when compared to Rubin's rules for different imputation sets. Finally, the authors back-transform their estimate and confidence interval bounds to the regular scale.

As @joe-74 pointed out in his comment, and Debray et al in the reference, it all depends on whether you would assume a normal distribution around the c-index estimate (or other estimate you'd want to pool) and finding a low variance of said estimate. This assumption is necessary in order to avoid the area of the (c-index) scale which is not normally distributed (e.g. near the 1.0 bound). Furthermore, assuming normality will result in a symmetric confidence interval, which suffers the same problems as the estimate itself (i.e. on the bounded scale of concordance between $[0,1]$). To make this clear, the difference in concordant pairs between c-indices of 0.75 and 0.80 is a smaller than the difference between 0.90 and 0.95.

Second, to me, the setting of pooling multiple studies or multiple imputation datasets does not matter (please comment if you think otherwise).

Consequently, while this may be opinionated (I still do not have a clear reference where the possible bias or error due to pooling without transformation was actually studied), I'd rather not assume this normality for a scale which is inherently non-normal ($[0,1]$ bounded).

On a side note, using this strategy, values bounded only on one side (such as the Observed:Expected ratio $[0,∞]$) can be transformed with a log transformation (as mentioned per the Siregar et al reference).

To conclude, I would recommend (myself) to logit-transform the c-indices found in the imputation datasets to an unbounded scale and apply Rubin's rules on these transformed values to pool them (including calculating the variance and the confidence interval). And finally, back-transforming the resulting values into the final pooled estimate.