5
$\begingroup$

Say I want to predict whether a patient develops a disorder or not. I have two prediction 'models': Clinicians estimating the probability of a patient developing the disorder and logistic regression using patient information as predictors.

What is the best way to compare both 'models', meaning which models makes the better predictions (both descriptively and inferential)? Is there a way to compare them by just using the probabilities given for each patient by both models? I have thought of using the human probabilities as input in a new logistic regression, but I'm am not sure whether this is a valid approach.

In my case, missclassifying an ill patient as healthy would be worse than missclassifying a healthy patient as ill - but as far as I understand classification this would be a question of my 'subjective' calibration (meaning the threshold for classification I choose), and can't be evaluated based only on predicted probabilities?

Improve this question
$\endgroup$
2
  • $\begingroup$ Do you want a test, or just a descriptive statistic? More generally, what would you want to be able to say about the comparison? $\endgroup$
    – gung - Reinstate Monica
    Commented Mar 22, 2021 at 17:20
  • $\begingroup$ Thank you, I edited the question, hope it's more specific now. $\endgroup$
    – Robn
    Commented Mar 25, 2021 at 8:51

2 Answers 2

Reset to default
2
$\begingroup$

Stephan makes good, although brief, points. Assuming the clinician is giving a probability and not a prediction of the form "This person will die, this person will not" there are a couple ways to compare the two models.

  • Brier Score: The brier score is a quadratic scoring rule. It is the mean squared error between the observed outcomes and the probabilities each method assigns to those outcomes. Lower is better in this case.

  • Calibration: If you assign a probability of 75% to 100 subjects, then 75 out of the 100 subjects should get the outcome. A proabbalistic model which is poorly calibrated gives probabilities which are either too extreme or not extreme enough. Neither are preferable. You want a well calibrated model.

  • Discrimination: See area under the receiver operating characteristic curve. This metric can be insensitive to actual improvements from model to model and so a lack of improvement here does not necessarily mean the alternative model did not improve the prediction quality. However, from what I know about how humans understand probability, I'm fairly confident a logistic regression would show improvement in this particular metric over a clinician.

  • R squared: There are several ways to compute r squared for probabilistic models. I recommend Nagelkerke's r squared.

Many of these are discussed in Frank Harrell's book Regression Modelling Strategies. I recommend taking a look at that book.

Improve this answer
$\endgroup$
4
  • $\begingroup$ This tells the OP things to use, but doesn't quite tell them how "to compare both 'models'". Can you make this into a more complete answer? $\endgroup$
    – gung - Reinstate Monica
    Commented Mar 22, 2021 at 16:40
  • $\begingroup$ @gung-ReinstateMonica I think these metrics are good modes of comparison, and any additional information I would add would be "lower/higher scores are better". A full discussion about how to compare models would be too long for me to write, and so I'm willing to point OP to references at the very most. $\endgroup$
    – Demetri Pananos
    Commented Mar 22, 2021 at 16:55
  • $\begingroup$ Thanks, that helped a lot! Not sure I understand calibration correctly. Calibration is not a metric for model comparison like Brier score and the rest you mention, right? You're simply saying that any classification model must be well calibrated? $\endgroup$
    – Robn
    Commented Mar 25, 2021 at 8:56
  • $\begingroup$ @Robn Calibration is a type of model comparison. It is a measure of how well the predicted probabilities compare to observed probabilities. Broadly, you want a well calibrated model, and so comparing models on calibration is something one might do. $\endgroup$
    – Demetri Pananos
    Commented Mar 25, 2021 at 14:15
1
$\begingroup$

You have two probabilistic classification models. Assuming you have the ground truth, you can assess the quality of your models using proper scoring rules. The tag wiki contains information and pointers to literature.

Improve this answer
$\endgroup$
4
  • $\begingroup$ This seems like it should probably be a comment. $\endgroup$
    – gung - Reinstate Monica
    Commented Mar 22, 2021 at 16:41
  • $\begingroup$ @gung-ReinstateMonica: See here for a motivation for short answers. Longer answers are always welcome. Demetri gives a better and longer answer, which I'll happily upvote now. $\endgroup$
    – Stephan Kolassa
    Commented Mar 22, 2021 at 16:42
  • $\begingroup$ I'm aware of that (& I had upvoted it). It's a judgment call, but this seems more like a comment to me. I respect your right to disagree, of course... $\endgroup$
    – gung - Reinstate Monica
    Commented Mar 22, 2021 at 17:18
  • 1
    $\begingroup$ @gung-ReinstateMonica: certainly, and I have no problem with your believing this should have been a comment. It's just that I see many, many questions that do not get a better answer, so the choice is between an "answer-comment" and a "comment-answer". I'd rather have the second. Also, to be honest, I do try to fill some tag wikis with information and pointers to literature precisely for cases like this, so I can point people to them, since we don't have a canonical duplicate for the many questions that are very close cousins. $\endgroup$
    – Stephan Kolassa
    Commented Mar 22, 2021 at 18:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.