4
$\begingroup$

I built a really good Random Forest classifier model for predicting whether or not someone will lose weight using several biomarkers as predictors. When I try to build a Random Forest regression model for predicting how much weight someone will lose/gain using the same predictors, it's really really bad.

I am using adjusted R2 to asses the regression model and AUROC to asses the classifier model. Is this disconnect conceptually possible or does this mean I must have messed up somewhere?

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ How do you decide on the parameter count when you use adjusted $R^2$ with a random forest model? $\endgroup$
    – Dave
    Commented Dec 27, 2021 at 22:13
  • 1
    $\begingroup$ When you calculate adjusted $R^2$, how are you calculating it? $\endgroup$
    – Dave
    Commented Dec 27, 2021 at 22:54
  • 1
    $\begingroup$ It is conceptually possible - classification is a harder problem than regression. However, it is more likely that your optimisation is not so effective for regression (precisely because its a harder problem). maybe you need a higher tree depth etc/need to change parameter you are not optimising over. perhaps a useful sense check is calculating the auroc of your regression model. $\endgroup$
    – seanv507
    Commented Dec 27, 2021 at 22:59
  • 1
    $\begingroup$ How do you determine $p$ in your random forest model? $\endgroup$
    – Dave
    Commented Dec 27, 2021 at 23:33
  • 2
    $\begingroup$ in my above comment, I meant to write " classification is an easier problem than regression" $\endgroup$
    – seanv507
    Commented Dec 27, 2021 at 23:56

1 Answer 1

Reset to default
14
$\begingroup$

Yes this is not just possible, but likely. You are answering two different questions with your models.

  1. Whether or not they gain/lose weight - classification
  2. How much weight they gain/lose - regression

The first question is about the 'sign' of the weight loss (+ or -) while the second is about the 'sign' and 'magnitude'. The discrepancy between how well your models do suggest you can predict the 'sign' but not the 'magnitude'.

One way to think about this: Imagine if people with biomarker 1 lose: 0.1, 1, 10, 100 lbs while those without it gain 0.1, 1, 10, or 100 lbs. You can see that a classifier works great, but the constraint on the weight loss is extremely poor because of the huge variability between subjects.

So yes this is conceptually possible and does not indicate an error in your modeling (though obviously it doesn't rule that out either).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.