10
$\begingroup$

I'm using 100 times 10-fold repeated cross-validation to assess the ROC-AUC performance improvement of adding a biomarker to an existing model: Model_A : pred1 + pred2 Model_B :pred1 + pred2 + pred3

I've seen advice before to use the Wilcoxon rank test to compare the AUCs between each fold. Averaging ROC curves over folds in cross-validation

Should I pull the median p-value from this? Is it acceptable to use the diff.resamples function in the Caret package and use the Wilcoxin rank instead of the default t-test? Does it need Bonferonni correction if only looking at AUC? https://www.rdocumentation.org/packages/caret/versions/6.0-86/topics/diff.resamples

Lastly, any thoughts on using DeLong or the likelihood ratio test. instead?

Improve this question
$\endgroup$
4
  • $\begingroup$ The answer might depend on the nature of your model. Is this a logistic regression or something more complicated like a support-vector machine or a boosted tree? $\endgroup$
    – EdM
    Sep 9, 2020 at 18:32
  • $\begingroup$ Logistic regression models. A collaborator took the AUCs from the median fold of Model A vs the median fold of Model B and compared with De Long test. I don't think de Long should be used for nested models and it's difficult to compare crossvalidated data. Any thoughts? ncbi.nlm.nih.gov/pmc/articles/PMC3684152 $\endgroup$
    – StephenD
    Sep 9, 2020 at 21:12
  • $\begingroup$ The paper that you linked says, near the end of the Background: "the DeLong et al. method is designed for comparing two fixed models that are tested on a common dataset independent of the training set." As I read it, the de Long method works OK with nested models if that requirement is met, but it's not clear how it would properly be applied to cross-validated models. Note that the required separate train/test data split needs many more cases than is typical of biomarker studies; see this page and its links for example. $\endgroup$
    – EdM
    Sep 10, 2020 at 17:02
  • $\begingroup$ Thanks Ed, yes I agree. Comparing the "average" folds is likely to lead to an unpaired test and thus a more conservative estimate. As far as I'm aware methods to compare cross-validations/ bootstraps don't yet exist. If they did they could potentially replace the t-test etc $\endgroup$
    – StephenD
    Sep 10, 2020 at 19:33

3 Answers 3

Reset to default
1
$\begingroup$

As these are nested logistic regression models there is no doubt that Frank Harrell's comment shows how to proceed: do the standard likelihood ratio test on the 2 models,* based on all of the data, to determine whether adding the third predictor improves performance. That has a well established theoretical basis, is more sensitive for detecting model differences than AUC, and it doesn't inherently require cross validation.

Cross validation or bootstrapping to evaluate model optimism and calibration would certainly help bolster your case that your modeling approach is correct, but the emphasis shouldn't be on AUC. There's no harm in showing how much the AUC changes, but that should be a secondary consideration. The validate function in Harrell's rms package provides several measures of model quality based on bootstrapping or cross validation, including a Dxy rank-correlation value (both original and optimism-corrected) that can be transformed into an AUC value.

*I'm a bit worried that you seem to be including so few predictors in your model. Logistic regression can have an omitted-variable bias if a predictor associated with outcome is left out of the model. Unlike linear regression, the omitted predictor doesn't even need to be correlated with the included predictors to get biased estimates. That's not to say you should be overfitting, but there are usually so many clinical variables associated with some condition or outcome that only including 2 or 3 would tend to be risky.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks @EdM , any thoughts on this paper? I suspect it used ideal conditions ncbi.nlm.nih.gov/pmc/articles/PMC3733611/#!po=0.735294 Yes, it's a small number of clinical variables but we would consider it more a pilot/ exploratory study. $\endgroup$
    – StephenD
    Sep 10, 2020 at 10:18
  • $\begingroup$ Why not just use lasso or ridge and cross-validate? $\endgroup$
    – Estimate the estimators
    Sep 27 at 7:29
  • $\begingroup$ @Estimatetheestimators the OP was about a very simple model with no more than 3 predictors, so ridge/lasso wouldn't be of much benefit in that case. Also, inference (confidence intervals, p-values) can be tricky with ridge/lasso. Those approaches often are useful, as you suggest, particularly when there are more predictors than you can include without overfitting. For such situations you should also look at Frank Harrell's Regression Modeling Strategies, particularly Chapter 4, for "data reduction" strategies that often draw on subject-matter knowledge. $\endgroup$
    – EdM
    Sep 27 at 13:16
8
$\begingroup$

Instead of averaging AUCs per fold you can calculate two ROC curve per iteration for Model_A and Model_B (since every instance is exactly predicted once in k-fold CV). To calculate whether the addition of a biomarker results in a model with significantly different AUC you can use DeLong's test. Here, I wouldn't use the median of the p-values - a simple count will do (e.g: around 5 significant p-values out of 100 times 10-fold CV can be explained by chance and indicate no improvement in model performance).

Different approaches to "combine" your p-values are mentioned in "Statistical Methods for Meta-Analysis" by Larry V. Hedges and Ingram Olkin.

If you are using Python and want to use DeLong's test, this blog post might be helpful (altough still in draft): https://biasedml.com/roc-comparison/

Improve this answer
$\endgroup$
7
  • 5
    $\begingroup$ Better yet, you can evaluate the log loss, Brier score, or another proper scoring rule! stats.stackexchange.com/questions/210700/… $\endgroup$
    – Dave
    Sep 9, 2020 at 10:57
  • 4
    $\begingroup$ AUROC (c-index; concordance probability; Mann-Whitney $U$-statistic) is not sensitive enough for comparing competing models. See alternatives here. $\endgroup$
    – Frank Harrell
    Sep 9, 2020 at 11:51
  • $\begingroup$ Hi @Laksan Nathan, Thanks for the answer. I think the absence of an actual p-value would lead to information loss. I'm unclear as to whether I can use any statistical test across cross-validation folds. From what I've read it violates the principles of the t-test/wilcoxon rank machinelearningmastery.com/… . $\endgroup$
    – StephenD
    Sep 9, 2020 at 14:06
  • $\begingroup$ @Frank Harrell . Thank you, I think I will look to use the log rank test for the main statistical test. I'll also have cross-validated AUCs as these are useful but I don't think it's possible to compare them easily across folds and it reduces the power of the comparison. A collaborator used the AUCs from the median fold of Model A vs the median fold of Model B but I don't think that's an accurate way.. it uses an unpaired De Long's which would underestimate the difference $\endgroup$
    – StephenD
    Sep 9, 2020 at 14:11
  • 3
    $\begingroup$ Note that this has nothing to do with the log rank test. This is the model-based likelihood ratio $\chi^2$ test. $\endgroup$
    – Frank Harrell
    Sep 9, 2020 at 16:10
0
$\begingroup$

DeLong’s test may not be suitable for nested model comparisons (e.g. https://pubmed.ncbi.nlm.nih.gov/22415937/). You may consider R^2 based test (https://www.sciencedirect.com/science/article/pii/S0002929723000046), which is available in CRAN (https://cran.r-project.org/web/packages/r2redux/index.html).

For your model comparisons, there are two ways.

Model_A : y = pred1 + pred2 + e Model_B : y = pred1 + pred2 + pred3 + e

mod = lm (y ~ pred1 + pred2 + pred3) merged_predictor = cbind(pred1, pred2) %*% mod$coefficients[2:3]

Then, the comparison can be equivalently expressed as Model_A : y = merged_predictor + e Model_B : y = merged_predictor + pred3 + e

Then, you can use r2redux as r2_diff(dat,c(1,2),1,nv) (please see manual in https://cran.r-project.org/web/packages/r2redux/index.html).

Note that whether using lm or glm with logit link wouldn’t be really matter, i.e. the result would be very similar.

Secondly, you can use preadjusted y as Model_B : y* = pred3 + e where y* is adjusted y for pred1 and pred2, i.e. y* = mod2$residuals with mod2 = lm(y ~ pred1 + pred2).

Then, you can use r2redux as r2_var(dat,1,nv) with properly specified inputs (please see manual in https://cran.r-project.org/web/packages/r2redux/index.html).

Additional note. There is one-to-one mapping between R^2 and AUC (https://cran.r-project.org/web/packages/R2ROC/index.html), which can also be used in such model comparisons. Please see https://www.biorxiv.org/content/10.1101/2023.08.01.551571v1.full.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Welcome to Cross Validated! Could you please explain the relationship between AUC and $R^2?$ How are you calculating $R^2?$ If you just consider squared correlation between predicted and true values, the calculation can miss some major issues, and I would be weary of such a calculation, preferring to calculate $R^2$ as a model comparison. I discuss those issues here with demonstrations here. $\endgroup$
    – Dave
    Aug 24 at 10:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.