2
$\begingroup$

The Boruta algorithm is a wrapper built around the random forest classification algorithm. It tries to capture all the important, interesting features in data with respect to an outcome variable.

I'm using boruta to select features before running random forest classifier. What I found is that the features selected is highly depend on the seeds I set. The selected features range from 5 to 8, depending on the seeds. I understand different seed might produce somehow different results. But my results vary quite a bit. Is this normal? If the result is so 'unstable', depend on the seeds, how I can justify my feature selection method?

Improve this question
$\endgroup$
4
  • $\begingroup$ Generally, random forest does not need pre-screened features to do a good job. It's probably better practice to grow a large forest (i.e. thousands of trees), see what features it did not use, then screen those out and refit the forest. $\endgroup$
    – Matthew Drury
    Oct 23, 2018 at 16:45
  • $\begingroup$ @Matthew Drury, When you screen out those features that it did not use and refit the data, you will have a new ranking of features, then if you again screen out the features it does not use(or least used), then refit the model, you will get another feature rankings, .... This is basically recursive feature selection does, right? $\endgroup$
    – zesla
    Oct 23, 2018 at 18:14
  • $\begingroup$ I mean, honestly, I don't think there's much point to screening out the features in the context of a random forest. The algorithm is designed to be robust to useless features. So I suppose a question is, what is the intent of your feature selection? $\endgroup$
    – Matthew Drury
    Oct 23, 2018 at 22:45
  • $\begingroup$ It may be that you don't really have good variables in your set, hence the algorithm pretty much chooses these at random, whichever comes first. $\endgroup$
    – user2974951
    Oct 24, 2018 at 5:39

1 Answer 1

Reset to default
1
$\begingroup$

For a fixed training set, you can lower the variance of your model by increasing the number of iterations in your ensemble (more trees).

If this does not help you should consider tuning the fraction of features to consider at each split, as well as size of bootstrap samples for each tree.

This should result in a more robust ensemble -> feature importance.

(Remark: So you are basically doing feature selection with RF before using RF?)

Improve this answer
$\endgroup$
4
  • $\begingroup$ anything wrong with feature selection with RF before RF? Could you elaborate a little bit? I see people do that all the time. random forest rfcv function does that job. $\endgroup$
    – zesla
    Oct 23, 2018 at 18:17
  • $\begingroup$ There are other algorithms that cannot give you insights on feature importances. So RF before that seem reasonable. But I have not seen RF before RF. Is there a reason why you assume that this works better than simply doing RF once? $\endgroup$
    – mlvalidated
    Oct 23, 2018 at 20:49
  • $\begingroup$ I'm using feature selection method that is relevant to the classifier. Do it make sense? Can you explain a little bit on why RF before RF is not good? what about running logistic regression after RF feature selection? Thanks for your advice $\endgroup$
    – zesla
    Oct 23, 2018 at 21:00
  • $\begingroup$ I am not saying that it is not good. Maybe that is the best way to deal with your problem (for whatever reason). If your experiments show better results this way (which I doubt), then you should stick to it. $\endgroup$
    – mlvalidated
    Oct 23, 2018 at 21:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.