0
$\begingroup$

I have a patient dataset on which I trained a RF classifier to predict whether a patient ends up in the hospital or not. Nevertheless, this dependent variable is imbalanced (66% of the patients ended up in hospital and 33% didn't).

I obtained an ROC-AUC score of ~76% which is not too bad, but when I look at the precision and recall (and the confusion matrix) there is something funny going on.

Precision = 0.66 & Recall = 1

Also, looking at the predicted labels, the RF classifier predicted everyone of the test set to end up in the hospital (label 1). Is this because of the imbalance in the dependent variable (hospitalisation)? How to tackle this problem if this were the case?

Improve this question
$\endgroup$
5
  • $\begingroup$ So you've got a model, which just predicts a single number for everyone. That isn't a very good model. Can you tell us more about the data? What are you using to predict the outcome? How were the data collected $\endgroup$
    – Demetri Pananos
    Commented Jan 26, 2021 at 18:02
  • $\begingroup$ @DemetriPananos Yes exactly, not very useful either. I use all kinds of clinical features that are available in a health record (age, blood pressure, sodium levels etc.). $\endgroup$
    – JonnDough
    Commented Jan 26, 2021 at 18:14
  • $\begingroup$ Why don't we start a little simpler and just do a logistic regression first. How does that perform? You can then relax the assumption of linearity using splines $\endgroup$
    – Demetri Pananos
    Commented Jan 26, 2021 at 18:36
  • $\begingroup$ @DemetriPananos I ran a logistic regression as well and got similar results. Excuse my ignorance but what are splines? I can't seem to find it in SciKit learn's logistic regression implementation. $\endgroup$
    – JonnDough
    Commented Jan 26, 2021 at 18:41
  • 1
    $\begingroup$ Splines are just a different basis function. We can use them to relax linearity. If a logistic regression had similar results, I don't think I can help much without seeing the data. Sorry $\endgroup$
    – Demetri Pananos
    Commented Jan 26, 2021 at 18:45

1 Answer 1

Reset to default
2
$\begingroup$

Handling the imbalance is worth trying; the usual techniques are reviewed here: https://www.kdnuggets.com/2017/06/7-techniques-handle-imbalanced-data.html.

Before doing that, consider changing the cutoff for class probability from its (presumed) default of 0.5. In other words, predict hospitalization only if the P(hosp) returned by the random forest model is greater than (for instance), 0.75 (and predict nonhospitalization otherwise), and see how that affects your confusion matrix. If you're using sklearn.ensemble.RandomForestClassifier, call the predict_proba method rather than predict.

And if none of these help, it may be that your input features are simply not predictive for hospitalization.

Improve this answer
$\endgroup$
9
  • $\begingroup$ Thank you for your input! How do I change this cut-off in the RF model? $\endgroup$
    – JonnDough
    Commented Jan 26, 2021 at 18:12
  • $\begingroup$ What software are you using? $\endgroup$
    – Mark Pundurs
    Commented Jan 26, 2021 at 18:14
  • $\begingroup$ Oh sorry for not stating that. I am using Python and used SciKit-learn's implementation of a RF. $\endgroup$
    – JonnDough
    Commented Jan 26, 2021 at 18:14
  • 1
    $\begingroup$ No, the cutoff for predict_proba is not related to any prior estimate of any probability; that cutoff is simply a hyperparameter of the model, like (for instance) max_features. (As such, its value should finally be chosen using cross-validation - but first do some simple exploration to see if varying that cutoff helps at all.) $\endgroup$
    – Mark Pundurs
    Commented Jan 26, 2021 at 18:39
  • 1
    $\begingroup$ I see. I'll read a bit on it. In the meanwhile I accepted your answer as most helpful and thank you for pointing me into the right direction! $\endgroup$
    – JonnDough
    Commented Jan 26, 2021 at 18:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.