3
$\begingroup$

I would like to compare a risk prediction test (model) with a single predictor (continuous variable):

Let's say I have a risk prediction testA (e.g. a logistic regression model) that gives a risk probability for cancer (yes/no) between 0-1 when it is applied on a patient with the characteristics B, C, D, E (test A uses all of these characteristics B-E).

Now there has been a new characteristic X found (a continuous variable such as systolic blood pressure) and we think that it could be a good predictor of the cancer state (yes/no) of a patient.

If I want to compare the testA and the characteristic X now to decide which is the better discriminator for the cancer state (yes/no), how could I do that?

I read through

  1. Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures. Epidemiology. 2010;21(1):128–138.

and thought about performing a ROC analysis and compare the AUC of the testA with the AUC of the characteristic X using a DeLong test.

However, since characteristic X ist not really model, I am not sure if my approach makes any sense.

Thank you for your help.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

In theory ROC analysis doesn't care if your test is a model or not. In fact, ROC analysis doesn't even see the model, only the model predictions. Therefore whether you have a model or not shouldn't matter.

In practice you should take into consideration the shape of the ROC curve. Especially if either testA or characteristicX has very few or even a single threshold, the ROC curve will be under-estimated, and the comparison might be misleading.

I have done similar analysis myself in a 2013 paper of mine, if you want to check it out, where I compare several measured biomarkers (characteristics XYZ) with several types of models that includes them (test ABC), in a cross-validation setting.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Dear Xavier. Thank you very much for sharing your paper. Do I understand it correctly that the test (i.e. the panel) would be rated as positive only if all parameters reached the treshold? An implementation of your method in R (probably with a multivariate discovery approach) would be of great interest for medical research. Have you done any net benefit analysis to compare your method to an approach using log. regression? Ps: I am a huge fan of your pROC package. $\endgroup$
    – captcoma
    Commented Jul 16, 2019 at 10:50
  • 1
    $\begingroup$ @captcoma no the final positivity is a parameter as well. See github.com/xrobin/PanelomiX $\endgroup$
    – Calimo
    Commented Jul 16, 2019 at 17:05
3
$\begingroup$

The $c$-index (concordance probability; AUROC; see also Somers' $D_{xy}$ rank correlation coefficient) is not sensitive enough for comparing two models. See http://fharrell.com/post/addvalue for the most sensitive measures.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.