I'm trying to classify if it will be rain tomorrow (1 for yes and 0 for no).
I've done pre-processing and ran a k-fold CV to choose the model with highest mean AUC score.
I received 3 models with same score (0.87) and I'm trying to decide which one of them to use.
The models are : Random Forests, ANN and Logistic Regression.
How can I decide which model is the best for me?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
If all models really do have the same k-fold AUC, you should choose the one which is easiest to interpret. This would be the model that makes the most real world sense, and ideally has as few predictors as possible.
Assuming you have enough data, you should try estimating your models on a training sample, then apply them to a testing sample to obtain an unbiased estimate of model performance. Further, you could consider which outcome is more important to you (either it rains or doesn't) and use what is known as partial AUC.
$\begingroup$
$\endgroup$
To add a bit to the answer from @ralph (+1), do consider whether AUC is the best measure to use. Although it's much better than many alternatives, a strictly proper scoring rule like log-loss or the Brier score can have better power to discriminate among models.
The point about focusing on the outcomes of most importance to you is quite important. AUC estimates performance over the entire range of probabilities, so it might not be what you need for your particular application and your tradeoffs between false positives and false negatives.
Finally, instead of choosing a single model you could consider combining information from all of them, as is done in the superlearner approach.
