I am using Random Forest Regression (with Python sklearn, but could easily switch to R if that would work better) to predict a variable. I think my model is starting to do fairly well, however I see a very clear pattern in the error: the residual error is highly correlated with distance from the mean value. Now I understand that that is somewhat the point of using a forest: to increase bias over variance. I can obviously increase variance again by reducing the number of trees that are used, but that also increases overall error. Is there a better way of doing this?

  • 1
    $\begingroup$ Good question! I wondered about the same problem in an answer here: stats.stackexchange.com/questions/66757/… $\endgroup$ – Flounderer Mar 16 '18 at 16:23
  • 1
    $\begingroup$ FWIW, correlation between residual and value is not "regression to the mean." Such a correlation rarely occurs because most models are flexible enough that they guarantee its absence. Are you sure this correlation exists? How have you measured it? $\endgroup$ – whuber Mar 16 '18 at 16:42
  • $\begingroup$ I checked visually. But went back and did a spearman correlation test: SpearmanrResult(correlation=-0.7704457013630613, pvalue=1.9859042248520221e-59) $\endgroup$ – Acrofales Mar 16 '18 at 17:06
  • $\begingroup$ @whuber I thought about it a bit more, and think I can reason out why it is happening. Firstly, even with the bagging approach from RF, I obtain a fairly weak overall regressor (r² of ~0.4), which is somewhat expected (noisy data), but could cause some trees to be very wrong and cause this "regression to the mean". Moreover, my target variable is not uniformly distributed over the range: I thought I might obtain a better result if I balance my training set. I'm not sure yet whether I should resample or use SMOTE. I am leaning to SMOTE, because I don't have a large data set to start with. $\endgroup$ – Acrofales Mar 19 '18 at 10:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.