I'm dealing with an imbalanced classification problem and I'd like to use feature importance from gradient boosting decision trees for recursive feature elimination in order to get rid of redundant features (on every iteration, remove feature with lowest importance, and calculate the resulting validation performance). I was planning on training neural networks later with the reduced feature set. The problem is that whenever I change my "seed" number (random state) when training trees using lightGBM, the feature importance rankings can change even quite significantly! What is a good way of dealing with this? Should I e.g. use 10 different seed numbers, and average the results?
I am using early stopping with a fixed validation subset, not cross-validation. Because of this I was surprised to see the large impact of the seed number. It is also making my hyperparameter search results inconsistent if I do not fix the seed number. I have 30-40 features and some 10**6 samples.