1
$\begingroup$

I'm dealing with an imbalanced classification problem and I'd like to use feature importance from gradient boosting decision trees for recursive feature elimination in order to get rid of redundant features (on every iteration, remove feature with lowest importance, and calculate the resulting validation performance). I was planning on training neural networks later with the reduced feature set. The problem is that whenever I change my "seed" number (random state) when training trees using lightGBM, the feature importance rankings can change even quite significantly! What is a good way of dealing with this? Should I e.g. use 10 different seed numbers, and average the results?

I am using early stopping with a fixed validation subset, not cross-validation. Because of this I was surprised to see the large impact of the seed number. It is also making my hyperparameter search results inconsistent if I do not fix the seed number. I have 30-40 features and some 10**6 samples.

Improve this question
$\endgroup$
2
  • $\begingroup$ 1. Can you please give us some metrics of how much the rankings change? 2. 30-40 features are well within the capabilities of a standard NN, there is little reason to use feature selection before that. 3. Given that the data you are working with is rather manageable size-wise ($10^6 \times 40$ is by no mean a prohibiting sample size) I would suggest focusing on getting your GBM "as good as it gets" and turn your attention to NN afterwards. $\endgroup$
    – usεr11852
    Aug 25, 2018 at 11:48
  • $\begingroup$ They change quite a bit. I decided to use a large random forest (N=1000) for feature importance rankings instead, I assume they should be more robust compared to GBM, since each tree is built independently. I have experimented with removing highly correlated features, the impact on performance seems very modest. However, I am still interested in the feature importance scores for interpreting the data. $\endgroup$
    – Peterukk
    Sep 16, 2018 at 13:37

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.