3
$\begingroup$

I've been trying to identify an optimal random forest model (regressor) by constructing a series of models each with different variable composition. After several experiments, i've noticed something that I couldn’t explain. One model with independent variables A, B, C, D, E gave me an OOB score of 70%, with variable D having the highest variable importance (considerably higher than the others). However, after I removed D leaving only four variables in the model, the OOB score increased to 75%. I implemented this in both R (with randomForest package) and python (with RandomForestRegressor within the sklearn library) and got the similar results. Can someone please explain why?

$\endgroup$
  • $\begingroup$ How large is your dataset? The difference of 70% to 75% could just be noise in the estimate. Noise on the order of 5% is expected if the test set size is around 350 points or smaller. $\endgroup$ – AaronDefazio Mar 29 '17 at 0:08
1
$\begingroup$

It is unusual but not impossible: I was able to get close to that scenario when I introduced a skew into the response variable (x) and let D 'specialize' in explaining only that part of the gradient enter image description here And even though its overall explanatory power is very low if it were a single linear predictor, it is the best overall predictor for that model in terms of %IncMSEenter image description here When it is removed, the second best predictor (weak but more predictive across the entire gradient, c in this example) dominates, while overall fit is nearly the same (72 vs. 75% in this randomly simulated example).enter image description here

That said, it may not be the only way to get this pattern and I could not match that % increase, so perhaps you could provide a reproducible example for a more definitive solution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.