Why is ROC curve used in assessing how 'good' a logistic regression model is?

Question

Suppose you fitted a logistic regression model and find out that it is well calibrated,that is, good agreement between observed outcomes and predictions. Also, suppose that it agrees with some training data as well.We are happy so far but...

I just don't get why someone would bother to check the ability of model predictions to discriminate between those with and those without the outcome. Why care about the area under the rock curve. Why does my model need to discriminate well?

Is it the case that a large area under the ROC curve implies a well calibrated model?

Answer 1

A logistic regression doesn't "agree" with anything because the nature of the outcome is 0/1 and the nature of the prediction is a continuous probability. Agreement requires comparable scales: 0.999 does not equal 1. One way of developing a classifier from a probability is by dichotomizing at a threshold. The obvious limitation with that approach: the threshold is arbitrary and can be artificially chosen to produce very high or very low sensitivity (or specificity). Thus, the ROC considers all possible thresholds.

A discriminating model is capable of ranking people in terms of their risk. The predicted risk from the model could be way off, but if you want to design a substudy or clinical trial to recruit "high risk" participants, such a model gives you a way forward. Preventative tamoxifen is recommended for women in the highest risk category of breast cancer as the result of such a study.

Discrimination != Calibration. If my model assigns all non-events a probability of 0.45 and all events a probability of 0.46, the discrimination is perfect, even if the incidence/prevalence is <0.001.