# Perfect recall but moderate precision due to imbalance?

I have a patient dataset on which I trained a RF classifier to predict whether a patient ends up in the hospital or not. Nevertheless, this dependent variable is imbalanced (66% of the patients ended up in hospital and 33% didn't).

I obtained an ROC-AUC score of ~76% which is not too bad, but when I look at the precision and recall (and the confusion matrix) there is something funny going on.

Precision = 0.66 & Recall = 1

Also, looking at the predicted labels, the RF classifier predicted everyone of the test set to end up in the hospital (label 1). Is this because of the imbalance in the dependent variable (hospitalisation)? How to tackle this problem if this were the case?

• So you've got a model, which just predicts a single number for everyone. That isn't a very good model. Can you tell us more about the data? What are you using to predict the outcome? How were the data collected Commented Jan 26, 2021 at 18:02
• @DemetriPananos Yes exactly, not very useful either. I use all kinds of clinical features that are available in a health record (age, blood pressure, sodium levels etc.). Commented Jan 26, 2021 at 18:14
• Why don't we start a little simpler and just do a logistic regression first. How does that perform? You can then relax the assumption of linearity using splines Commented Jan 26, 2021 at 18:36
• @DemetriPananos I ran a logistic regression as well and got similar results. Excuse my ignorance but what are splines? I can't seem to find it in SciKit learn's logistic regression implementation. Commented Jan 26, 2021 at 18:41
• Splines are just a different basis function. We can use them to relax linearity. If a logistic regression had similar results, I don't think I can help much without seeing the data. Sorry Commented Jan 26, 2021 at 18:45

Before doing that, consider changing the cutoff for class probability from its (presumed) default of 0.5. In other words, predict hospitalization only if the P(hosp) returned by the random forest model is greater than (for instance), 0.75 (and predict nonhospitalization otherwise), and see how that affects your confusion matrix. If you're using sklearn.ensemble.RandomForestClassifier, call the predict_proba method rather than predict.
• No, the cutoff for predict_proba is not related to any prior estimate of any probability; that cutoff is simply a hyperparameter of the model, like (for instance) max_features. (As such, its value should finally be chosen using cross-validation - but first do some simple exploration to see if varying that cutoff helps at all.) Commented Jan 26, 2021 at 18:39