logistic regression: compare performance to human prediction Say I want to predict whether a patient develops a disorder or not. I have two prediction 'models': Clinicians estimating the probability of a patient developing the disorder and logistic regression using patient information as predictors.
What is the best way to compare both 'models', meaning which models makes the better predictions (both descriptively and inferential)? Is there a way to compare them by just using the probabilities given for each patient by both models? I have thought of using the human probabilities as input in a new logistic regression, but I'm am not sure whether this is a valid approach.
In my case, missclassifying an ill patient as healthy would be worse than missclassifying a healthy patient as ill - but as far as I understand classification this would be a question of my 'subjective' calibration (meaning the threshold for classification I choose), and can't be evaluated based only on predicted probabilities?
 A: Stephan makes good, although brief, points.  Assuming the clinician is giving a probability and not a prediction of the form "This person will die, this person will not" there are a couple ways to compare the two models.

*

*Brier Score:  The brier score is a quadratic scoring rule.  It is the mean squared error between the observed outcomes and the probabilities each method assigns to those outcomes.  Lower is better in this case.


*Calibration:  If you assign a probability of 75% to 100 subjects, then 75 out of the 100 subjects should get the outcome.  A proabbalistic model which is poorly calibrated gives probabilities which are either too extreme or not extreme enough. Neither are preferable.  You want a well calibrated model.


*Discrimination:  See area under the receiver operating characteristic curve.  This metric can be insensitive to actual improvements from model to model and so a lack of improvement here does not necessarily mean the alternative model did not improve the prediction quality.  However, from what I know about how humans understand probability, I'm fairly confident a logistic regression would show improvement in this particular metric over a clinician.


*R squared:  There are several ways to compute r squared for probabilistic models.  I recommend Nagelkerke's r squared.
Many of these are discussed in Frank Harrell's book Regression Modelling Strategies.  I recommend taking a look at that book.
A: You have two probabilistic classification models. Assuming you have the ground truth, you can assess the quality of your models using proper scoring rules. The tag wiki contains information and pointers to literature.
