I want to train a binary classification model using some tree ensemble (either xgboost or random forests). My dataset has some 50 features, and I believe some of them are redundant (there's correlation between some features, among other things).
I am wondering what order should I perform the following two tasks
Optimize hyperparameters for the tree ensemble (e.g. number of trees, tree depth etc.) via $k$-fold cross validation
Recursive Feature Elimination: Fix hyperparameters and train a model. Remove features one at a time (after each model is trained) based on feature weights (lower weights => less importance). That is, train a model on $p$ features, remove feature $j$ with the lowest weight so now we have $p-1$ features, and repeat this until either i) the desired number of features are left or ii) the accuracy hits a lower threshold
What I am concerned about is if I remove features first and then optimize hyperparameters second, I may be loosing out on some complex feature + hyperparameter interactions. However if I do the converse, tune hyperparameters with all features, then remove features, I may end up with a dataset for which the hyperparameters we found are no longer optimal.
Is there a correct way / order to do this?