7
$\begingroup$

I'm using 100 times 10-fold repeated cross-validation to assess the ROC-AUC performance improvement of adding a biomarker to an existing model: Model_A : pred1 + pred2 Model_B :pred1 + pred2 + pred3

I've seen advice before to use the Wilcoxon rank test to compare the AUCs between each fold. Averaging ROC curves over folds in cross-validation

Should I pull the median p-value from this? Is it acceptable to use the diff.resamples function in the Caret package and use the Wilcoxin rank instead of the default t-test? Does it need Bonferonni correction if only looking at AUC? https://www.rdocumentation.org/packages/caret/versions/6.0-86/topics/diff.resamples

Lastly, any thoughts on using DeLong or the likelihood ratio test. instead?

Improve this question
$\endgroup$
4
  • $\begingroup$ The answer might depend on the nature of your model. Is this a logistic regression or something more complicated like a support-vector machine or a boosted tree? $\endgroup$ – EdM Sep 9 '20 at 18:32
  • $\begingroup$ Logistic regression models. A collaborator took the AUCs from the median fold of Model A vs the median fold of Model B and compared with De Long test. I don't think de Long should be used for nested models and it's difficult to compare crossvalidated data. Any thoughts? ncbi.nlm.nih.gov/pmc/articles/PMC3684152 $\endgroup$ – StephenD Sep 9 '20 at 21:12
  • $\begingroup$ The paper that you linked says, near the end of the Background: "the DeLong et al. method is designed for comparing two fixed models that are tested on a common dataset independent of the training set." As I read it, the de Long method works OK with nested models if that requirement is met, but it's not clear how it would properly be applied to cross-validated models. Note that the required separate train/test data split needs many more cases than is typical of biomarker studies; see this page and its links for example. $\endgroup$ – EdM Sep 10 '20 at 17:02
  • $\begingroup$ Thanks Ed, yes I agree. Comparing the "average" folds is likely to lead to an unpaired test and thus a more conservative estimate. As far as I'm aware methods to compare cross-validations/ bootstraps don't yet exist. If they did they could potentially replace the t-test etc $\endgroup$ – StephenD Sep 10 '20 at 19:33
1
$\begingroup$

As these are nested logistic regression models there is no doubt that Frank Harrell's comment shows how to proceed: do the standard likelihood ratio test on the 2 models,* based on all of the data, to determine whether adding the third predictor improves performance. That has a well established theoretical basis, is more sensitive for detecting model differences than AUC, and it doesn't inherently require cross validation.

Cross validation or bootstrapping to evaluate model optimism and calibration would certainly help bolster your case that your modeling approach is correct, but the emphasis shouldn't be on AUC. There's no harm in showing how much the AUC changes, but that should be a secondary consideration. The validate function in Harrell's rms package provides several measures of model quality based on bootstrapping or cross validation, including a Dxy rank-correlation value (both original and optimism-corrected) that can be transformed into an AUC value.

*I'm a bit worried that you seem to be including so few predictors in your model. Logistic regression can have an omitted-variable bias if a predictor associated with outcome is left out of the model. Unlike linear regression, the omitted predictor doesn't even need to be correlated with the included predictors to get biased estimates. That's not to say you should be overfitting, but there are usually so many clinical variables associated with some condition or outcome that only including 2 or 3 would tend to be risky.

Improve this answer
$\endgroup$
1
6
$\begingroup$

Instead of averaging AUCs per fold you can calculate two ROC curve per iteration for Model_A and Model_B (since every instance is exactly predicted once in k-fold CV). To calculate whether the addition of a biomarker results in a model with significantly different AUC you can use DeLong's test. Here, I wouldn't use the median of the p-values - a simple count will do (e.g: around 5 significant p-values out of 100 times 10-fold CV can be explained by chance and indicate no improvement in model performance).

Different approaches to "combine" your p-values are mentioned in "Statistical Methods for Meta-Analysis" by Larry V. Hedges and Ingram Olkin.

If you are using Python and want to use DeLong's test, this blog post might be helpful (altough still in draft): https://biasedml.com/roc-comparison/

Improve this answer
$\endgroup$
6
  • 4
    $\begingroup$ Better yet, you can evaluate the log loss, Brier score, or another proper scoring rule! stats.stackexchange.com/questions/210700/… $\endgroup$ – Dave Sep 9 '20 at 10:57
  • 3
    $\begingroup$ AUROC (c-index; concordance probability; Mann-Whitney $U$-statistic) is not sensitive enough for comparing competing models. See alternatives here. $\endgroup$ – Frank Harrell Sep 9 '20 at 11:51
  • $\begingroup$ Hi @Laksan Nathan, Thanks for the answer. I think the absence of an actual p-value would lead to information loss. I'm unclear as to whether I can use any statistical test across cross-validation folds. From what I've read it violates the principles of the t-test/wilcoxon rank machinelearningmastery.com/… . $\endgroup$ – StephenD Sep 9 '20 at 14:06
  • $\begingroup$ @Frank Harrell . Thank you, I think I will look to use the log rank test for the main statistical test. I'll also have cross-validated AUCs as these are useful but I don't think it's possible to compare them easily across folds and it reduces the power of the comparison. A collaborator used the AUCs from the median fold of Model A vs the median fold of Model B but I don't think that's an accurate way.. it uses an unpaired De Long's which would underestimate the difference $\endgroup$ – StephenD Sep 9 '20 at 14:11
  • 2
    $\begingroup$ Note that this has nothing to do with the log rank test. This is the model-based likelihood ratio $\chi^2$ test. $\endgroup$ – Frank Harrell Sep 9 '20 at 16:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.