I have a patient dataset on which I trained a RF classifier to predict whether a patient ends up in the hospital or not. Nevertheless, this dependent variable is imbalanced (66% of the patients ended up in hospital and 33% didn't).

I obtained an ROC-AUC score of ~76% which is not too bad, but when I look at the precision and recall (and the confusion matrix) there is something funny going on.

Precision = 0.66 & Recall = 1

Also, looking at the predicted labels, the RF classifier predicted everyone of the test set to end up in the hospital (label 1). Is this because of the imbalance in the dependent variable (hospitalisation)? How to tackle this problem if this were the case?

  • $\begingroup$ So you've got a model, which just predicts a single number for everyone. That isn't a very good model. Can you tell us more about the data? What are you using to predict the outcome? How were the data collected $\endgroup$ – Demetri Pananos Jan 26 at 18:02
  • $\begingroup$ @DemetriPananos Yes exactly, not very useful either. I use all kinds of clinical features that are available in a health record (age, blood pressure, sodium levels etc.). $\endgroup$ – User2341234234 Jan 26 at 18:14
  • $\begingroup$ Why don't we start a little simpler and just do a logistic regression first. How does that perform? You can then relax the assumption of linearity using splines $\endgroup$ – Demetri Pananos Jan 26 at 18:36
  • $\begingroup$ @DemetriPananos I ran a logistic regression as well and got similar results. Excuse my ignorance but what are splines? I can't seem to find it in SciKit learn's logistic regression implementation. $\endgroup$ – User2341234234 Jan 26 at 18:41
  • 1
    $\begingroup$ Splines are just a different basis function. We can use them to relax linearity. If a logistic regression had similar results, I don't think I can help much without seeing the data. Sorry $\endgroup$ – Demetri Pananos Jan 26 at 18:45

Handling the imbalance is worth trying; the usual techniques are reviewed here: https://www.kdnuggets.com/2017/06/7-techniques-handle-imbalanced-data.html.

Before doing that, consider changing the cutoff for class probability from its (presumed) default of 0.5. In other words, predict hospitalization only if the P(hosp) returned by the random forest model is greater than (for instance), 0.75 (and predict nonhospitalization otherwise), and see how that affects your confusion matrix. If you're using sklearn.ensemble.RandomForestClassifier, call the predict_proba method rather than predict.

And if none of these help, it may be that your input features are simply not predictive for hospitalization.

  • $\begingroup$ Thank you for your input! How do I change this cut-off in the RF model? $\endgroup$ – User2341234234 Jan 26 at 18:12
  • $\begingroup$ What software are you using? $\endgroup$ – Mark Pundurs Jan 26 at 18:14
  • $\begingroup$ Oh sorry for not stating that. I am using Python and used SciKit-learn's implementation of a RF. $\endgroup$ – User2341234234 Jan 26 at 18:14
  • $\begingroup$ I added this information to the answer. $\endgroup$ – Mark Pundurs Jan 26 at 18:17
  • 1
    $\begingroup$ No, the cutoff for predict_proba is not related to any prior estimate of any probability; that cutoff is simply a hyperparameter of the model, like (for instance) max_features. (As such, its value should finally be chosen using cross-validation - but first do some simple exploration to see if varying that cutoff helps at all.) $\endgroup$ – Mark Pundurs Jan 26 at 18:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.