1
$\begingroup$

I did gridsearch with corss-validation on a trainingset to search for best hyperparameters for a Random Forest Regressor. And indeed the best parameterset gives good results in cross-validation (R^2 ~ 86%, which is slightly better than my previous results). The results on my Testset are quite similar, so everything seems fine to me.

BUT: The Hyperparameter for the minimum of samples per leaf is set to 1, which often leads to massive overfitting. And indeed, when applying the model to the trainingset, R^2 is 99%.

Now I wonder wether this is a problem? Everywhere you read that you should avoid overfitting. And when I trained a neural net on the same dataset, I had to regularize by dropout, so that R^2 on Testset (e.g. 80%) and on Trainingset (e.g. 90%) could meet in the midlle at around 85%. But when regularizing the random forest, R^2 on Trainingset AND Testset decreases.

So is overfitting here a problem, when the Random Forest Regressor (that definitly overfits with 99% to 86%) does a good job in cross-validation and Testset anyways? Or what would you do?

Thank you for your answers!

Improve this question
$\endgroup$
3
  • $\begingroup$ I'd like to point you to this: datascience.stackexchange.com/a/6430 $\endgroup$
    – Bernhard
    Commented Jan 10, 2021 at 21:47
  • $\begingroup$ Thank you, your link led me to this post: stats.stackexchange.com/questions/162353/… where Andy Liaw (who maintains the R-Package randomForest) is quoted, that Random Forest has by design nearly no loss an Training-Data, so that you should compare the accuracy of cross-validation with accuracy on Testset. So in my case this means nearly zero overfitting, which is usually expected from Random Forest. $\endgroup$
    – Scrabyard
    Commented Jan 11, 2021 at 9:03
  • $\begingroup$ Great. If that answers your question fully please write an answer an consider accepting it in two days time so that others can see, that your problem is resolved: see stats.stackexchange.com/help/self-answer $\endgroup$
    – Bernhard
    Commented Jan 11, 2021 at 11:14

1 Answer 1

Reset to default
0
$\begingroup$

As quoted here: What measure of training error to report for Random Forests? Andy Liaw, who maintains the R-Package randomForest, states that Random Forests tend to nearly zero loss on trainingset and that this perfect prediction is "by design".

So to tell wether a model is overfitting, one has to compare the accuracy from e.g. cross-validation with accuracy on the independent testset.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Do not forget the built-in out-of-bag estimates. $\endgroup$
    – cbeleites
    Commented Jan 12, 2021 at 17:51
  • $\begingroup$ Of course, but the oob-scores are only available if Random Forest is build with bootstrapping. I had to train three Random Forest Regressors on three different datasets and in all cases gridsearch (and performance on testset) showed that in my case bootstrapping was less efficient, so randomization only came from limiting the number of features available per tree. $\endgroup$
    – Scrabyard
    Commented Jan 14, 2021 at 16:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.