Does good calibration surpass roughly met assumptions and mediocre discrimination?

Question

My question arises from my current task to develop a clinical prediction model using ordinal logistic regression (with rms), but applies to any kind of regression analysis.

1. The proportional odds assumption (difficult to assess...) is only roughly met. (I am aware of the continuation ratio approach).

2. Results of measures of discrimination (another difficult topic) are not overwhelming.

3. But the calibration plot is quite nicely.

So, do I have to worry about 1 and 2 or does the very good calibration surpass basic assumptions and discrimination?

Answer 1

"If a model has poor discrimination, no adjustment or calibration can correct the model. If discrimination is good, the predictor can be calibrated without sacrificing discrimination. Here, calibrating predictions means modifying them without changing their rank order"

Tutorial in biostatistics: Multi variable prognostic models : issue in developing models, evaluating assumptions and adequacy and measure and reducing errors. Harrell et al. 1996).

So, I would be inclined to say that poor discrimination, but good calibration is a problem. But this is also a strange result. If you have a high probability, such a 0.9, to be perfectly calibrated you would need to have an observed frequency of 90% for the case (for example, in this decile 90% of patients do have cancer), and this relationship needs to be true throughout. In order to achieve something like that, I am inclined to say you need good discrimination too (i.e. False positives may bring down the observed frequency at the high end and vice versa, thus resulting in poor calibration).

Maybe you need to make sure that your calibration is actually "good" and discrimination "bad".