3
$\begingroup$

I have close to 10,000 variables. I know how random forest/XGB picks number of variables randomly for building the tree. Also regularization takes care of significance of variable by its coefficient.

But do I still need to do EDA/uniariate/bivariate and everything to reduce the variables as the first step. If yes, why? As I understand that above algorithm will pick the important variable by itself and are called ML algorithm because it doesn't require manual intervention to that extent.

I have seen that of I give limited variables as input to my Random Forest, it may work better. Please explain.

$\endgroup$
4
$\begingroup$

as I understand that above algorithm will pick the important variable by itself and are called ML algorithm because it doesn't require manual intervention to that extent.

It's called ML because the algorithm learns by itself some difficult pattern by looking at lots of data, and learning when it gets the wrong prediciton, working towards the goal of a good performance overall.

You still need to point it in the right direction. Or at least watch closely what it "learns". It's not magic, thankfully.

A good process analysis process always starts with some EDA, because you learn for example which variables are more interesting in terms of what's related to the outcome and what's not.

Those algorithms are simply "not good enough" to simply discard the information that they receive when they see lots of noisy variables. Thankfully there's still a need for a Data Scientist afterall.

RF and XGB tell you that some variables have low importance, that's correct, but because they use randomization during the process they still might build a tree that uses all noisy variables (not correlated with the outcome) plus one good variable (highly correlated).

This is somewhat related with the curse of dimensionality too.

So, no, the algorithm won't pick the important variables by itself, it will use everything you throw in the pot to achieve it's goal (high performance). But it will get lost if the hypothesis space is too damn big (which it is for 10k variables).

And it won't reach its full potential, which is reached via feature engeneering and feature selection, even after a first model where you give all your variables (for example Lasso followed by Random Forest).

$\endgroup$
1
$\begingroup$

You do not need to reduce the variables for RF / XGB as these methods work even with a huge number of variables and select the important ones based on importance scores. However, because of the curse of dimensionality, it may be better to remove noisy variables as the models should perform better in this case, although it's not guaranteed.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.