I've been trying to identify an optimal random forest model (regressor) by constructing a series of models each with different variable composition. After several experiments, i've noticed something that I couldn’t explain. One model with independent variables A, B, C, D, E gave me an OOB score of 70%, with variable D having the highest variable importance (considerably higher than the others). However, after I removed D leaving only four variables in the model, the OOB score increased to 75%. I implemented this in both R (with randomForest package) and python (with RandomForestRegressor within the sklearn library) and got the similar results. Can someone please explain why?
It is unusual but not impossible: I was able to get close to that scenario when I introduced a skew into the response variable (x) and let D 'specialize' in explaining only that part of the gradient And even though its overall explanatory power is very low if it were a single linear predictor, it is the best overall predictor for that model in terms of %IncMSE When it is removed, the second best predictor (weak but more predictive across the entire gradient, c in this example) dominates, while overall fit is nearly the same (72 vs. 75% in this randomly simulated example).
That said, it may not be the only way to get this pattern and I could not match that % increase, so perhaps you could provide a reproducible example for a more definitive solution.