Analysis of Feature Importances when features are dependent on one another

I can use random forests to determine which features are important when doing a prediction problem; for example.

< height, weight, IQ measure> -> Is considered obese?

Applying random forests could tell me that weight contributes 0.75 percent to obesity.

But what if weight was dependent on height? For example weight = height^2 + IQ Measure^3/height or something?

How would this affect the generated feature importances? Would this 'hidden' relationship have any consequences?

I just thought about this, and was wondering whether this could be an issue. As in, my feature importances would be inaccurate.

1 Answer

It is quite possible that it is an issue e.g. http://www.biomedcentral.com/1471-2105/9/307 the severity of which you can examine using recursive feature elimination http://arxiv.org/abs/1310.5726 but it is less of an issue than in multiple regression as was discussed here earlier Do correlated and/or derived fields require special consideration when using Random Forest?

• From my experience, random forests seems to give me more realistic feature importances. - But in theory should RFE do better? Would you always recommend RFE over RF?
– user46925
Nov 30, 2014 at 18:14
• The suggested rfe reference is just a way to explore how much it matters in this particular case, there are other ways. In my experience, RF is very robust to this problem (so I certainly don't recommend supplanting it), but as the first reference shows it is not always immune. And another consideration is why correlated predictors were included: whether they are necessary and meaningful in the context of your model or whether condensing or more carefully chosing them would be more justified. Nov 30, 2014 at 18:39