I am asking myself if it is a good idea to remove those variables with a negative variable importance value ("%IncMSE") in a regression context. And if it gives me a better prediction? What do you think?
Variable importance in Random forest is calculated as follows:
- Initially, MSE of the model is calculated with the original variables
Then, the values of a single column are permuted and the MSE is calculated again. For example, If a column (Col1) takes the values 1,2,3,4, and a random permutation of the values results in 4,3,1,2. This results in an MSE1. Then an increase in the MSE, i.e., MSE1 - MSE, would signify the importance of the variable.
We expect the difference to be positive, but in the cases of a negative number, it denotes that the random permutation worked better. It can be inferred that the variable does not have a role in the prediction,i.e, not important.
Hope this helps!
Please refer to the following link for a elaborated explanation!
This may be just a random fluctuation (for instance if you have small ntree).
If not, it may show that you have some serious amount of paradoxes in your data, i.e. pairs of objects with almost identical predictors and very different outcome. In this case, I would check twice if the model actually makes any sense and start thinking how I could get more attributes to resolve them.