I am using Random Forest Regression (with Python sklearn, but could easily switch to R if that would work better) to predict a variable. I think my model is starting to do fairly well, however I see a very clear pattern in the error: the residual error is highly correlated with distance from the mean value. Now I understand that that is somewhat the point of using a forest: to increase bias over variance. I can obviously increase variance again by reducing the number of trees that are used, but that also increases overall error. Is there a better way of doing this?
$\begingroup$
$\endgroup$
4
-
1$\begingroup$ Good question! I wondered about the same problem in an answer here: stats.stackexchange.com/questions/66757/… $\endgroup$– FloundererCommented Mar 16, 2018 at 16:23
-
1$\begingroup$ FWIW, correlation between residual and value is not "regression to the mean." Such a correlation rarely occurs because most models are flexible enough that they guarantee its absence. Are you sure this correlation exists? How have you measured it? $\endgroup$– whuber ♦Commented Mar 16, 2018 at 16:42
-
$\begingroup$ I checked visually. But went back and did a spearman correlation test: SpearmanrResult(correlation=-0.7704457013630613, pvalue=1.9859042248520221e-59) $\endgroup$– AcrofalesCommented Mar 16, 2018 at 17:06
-
$\begingroup$ @whuber I thought about it a bit more, and think I can reason out why it is happening. Firstly, even with the bagging approach from RF, I obtain a fairly weak overall regressor (r² of ~0.4), which is somewhat expected (noisy data), but could cause some trees to be very wrong and cause this "regression to the mean". Moreover, my target variable is not uniformly distributed over the range: I thought I might obtain a better result if I balance my training set. I'm not sure yet whether I should resample or use SMOTE. I am leaning to SMOTE, because I don't have a large data set to start with. $\endgroup$– AcrofalesCommented Mar 19, 2018 at 10:59
Add a comment
|