Improving a forest model by dropping features below a percent importance threshold? I'm wondering if there is a term for this process, where I can find more reading/information about it, and if this is valid or hacky overall. I've only used this process for tree-based models which have easily accessible feature importances (random forest, xgboost, etc.)
The process is as follows: I train a preliminary model with all the features, and track each feature's % importance. I then loop through different thresholds (e.g. 0%, 2%, 4%...) and drop all the features, from the original model, that have a percent importance below the threshold. I then choose the model, features, and threshold that yield the best accuracy, since that is what I am interested in for this particular case.
Is this a known approach, and is there anything wrong with it? Where can I find more information about it?
 A: In theory, you could take such an approach, but there's a number of caveats.
Firstly, whether this is helpful for predictive performance will be rather case specific and it could certainly make your model performance worse on new data. E.g. you could imagine that it would be helpful in case you think that there could be a number of predictors that have no effect at all, while if all predictors may have some potentially useful information, it may heavily depend on the amount of data you have, how correlated predictors are and various other factors. Note that correlation between predictors might be a serious issue for your proposed approach (e.g. if you have many very closely related variables, you could end up with each having lowish feature importance for each, but together they are very important to the model).
Secondly, how do you decide what's a good threshold value to use? You definitely would not want to rely on performance on training data, but rather you'd want to pick the threshold e.g. through cross-validation. I.e. it is like any other hyperparameter of your algorithm (and note: other hyperparameters may have different optimal values after removing this variable, which is e.g. obvious for the proportion of features to randomly pick for each tree and/or split within a tree).
Thirdly, this will affect the interpretation of the final model. Given the models you talk about, I'm assuming you're primarily interested in predictive performance, but just to be clear: interpreting just the final created model may be rather tricky due to the preceding model selection. In this context, it's useful to be aware that the variable selection might be a pretty variable in its outcome. If you just do the same thing a few times with different random number seeds, you might get different results, but that still understates it, to get a fuller picture bootstrapping the data and looking how much additional variability that adds in what variables get selected can provide useful insights. For many model selection procedures it turns out that it's rather variable (hey, maybe you'd even get better results by doing this repeatedly and averaging the resulting models) and thus, we should not pretend that such a selection procedure truly selects the variables that are really important. On the other hand, a "well, we can't know whether other variables are maybe important, but we're predicting decently well with these" interpretation is more reasonable.
