I've been trying to identify an optimal random forest model (regressor) by constructing a series of models each with different variable composition. After several experiments, i've noticed something that I couldn’t explain. One model with independent variables A, B, C, D, E gave me an OOB score of 70%, with variable D having the highest variable importance (considerably higher than the others). However, after I removed D leaving only four variables in the model, the OOB score increased to 75%. I implemented this in both R (with randomForest package) and python (with RandomForestRegressor within the sklearn library) and got the similar results. Can someone please explain why?
1
-
$\begingroup$ How large is your dataset? The difference of 70% to 75% could just be noise in the estimate. Noise on the order of 5% is expected if the test set size is around 350 points or smaller. $\endgroup$– AaronDefazioCommented Mar 29, 2017 at 0:08
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
It is unusual but not impossible: I was able to get close to that scenario when I introduced a skew into the response variable (x) and let D 'specialize' in explaining only that part of the gradient 
And even though its overall explanatory power is very low if it were a single linear predictor, it is the best overall predictor for that model in terms of %IncMSE
When it is removed, the second best predictor (weak but more predictive across the entire gradient, c in this example) dominates, while overall fit is nearly the same (72 vs. 75% in this randomly simulated example).
That said, it may not be the only way to get this pattern and I could not match that % increase, so perhaps you could provide a reproducible example for a more definitive solution.
