I would like to compare a risk prediction test (model) with a single predictor (continuous variable):

Let's say I have a risk prediction testA (e.g. a logistic regression model) that gives a risk probability for cancer (yes/no) between 0-1 when it is applied on a patient with the characteristics B, C, D, E (test A uses all of these characteristics B-E).

Now there has been a new characteristic X found (a continuous variable such as systolic blood pressure) and we think that it could be a good predictor of the cancer state (yes/no) of a patient.

If I want to compare the testA and the characteristic X now to decide which is the better discriminator for the cancer state (yes/no), how could I do that?

I read through

  1. Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures. Epidemiology. 2010;21(1):128–138.

and thought about performing a ROC analysis and compare the AUC of the testA with the AUC of the characteristic X using a DeLong test.

However, since characteristic X ist not really model, I am not sure if my approach makes any sense.

Thank you for your help.


In theory ROC analysis doesn't care if your test is a model or not. In fact, ROC analysis doesn't even see the model, only the model predictions. Therefore whether you have a model or not shouldn't matter.

In practice you should take into consideration the shape of the ROC curve. Especially if either testA or characteristicX has very few or even a single threshold, the ROC curve will be under-estimated, and the comparison might be misleading.

I have done similar analysis myself in a 2013 paper of mine, if you want to check it out, where I compare several measured biomarkers (characteristics XYZ) with several types of models that includes them (test ABC), in a cross-validation setting.

  • $\begingroup$ Dear Xavier. Thank you very much for sharing your paper. Do I understand it correctly that the test (i.e. the panel) would be rated as positive only if all parameters reached the treshold? An implementation of your method in R (probably with a multivariate discovery approach) would be of great interest for medical research. Have you done any net benefit analysis to compare your method to an approach using log. regression? Ps: I am a huge fan of your pROC package. $\endgroup$
    – captcoma
    Jul 16 '19 at 10:50
  • 1
    $\begingroup$ @captcoma no the final positivity is a parameter as well. See github.com/xrobin/PanelomiX $\endgroup$
    – Calimo
    Jul 16 '19 at 17:05

The $c$-index (concordance probability; AUROC; see also Somers' $D_{xy}$ rank correlation coefficient) is not sensitive enough for comparing two models. See http://fharrell.com/post/addvalue for the most sensitive measures.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.