Say I want to predict whether a patient develops a disorder or not. I have two prediction 'models': Clinicians estimating the probability of a patient developing the disorder and logistic regression using patient information as predictors.
What is the best way to compare both 'models', meaning which models makes the better predictions (both descriptively and inferential)? Is there a way to compare them by just using the probabilities given for each patient by both models? I have thought of using the human probabilities as input in a new logistic regression, but I'm am not sure whether this is a valid approach.
In my case, missclassifying an ill patient as healthy would be worse than missclassifying a healthy patient as ill - but as far as I understand classification this would be a question of my 'subjective' calibration (meaning the threshold for classification I choose), and can't be evaluated based only on predicted probabilities?