# Necessary to train, tune and test if only estimating variable importance?

In medicine the use of regression models may differ slightly from other fields. We usually build regression models from theory and subject matter knowledge. We're typically interested in estimating the effect of some clinical feature, while accounting for other variables. Less often, we develop more typical prediction models and in those circumstances we do respect the fact that one must train, tune and test models by conventional means.

However, I'm looking into tree based algorithms (random forest and GBM) and was thinking about this. I'm estimating the effect of a particular treatment, and I'm interested in the relative importance of that treatment. I'm not interested in predicting per se. As compared with regression models, tree based methods seem to have an inherent ability to provide estimates of relative variable importance, which is beautiful.

Can I apply the same thinking in machine learning? Can I skip the training and test procedure, fine tuning etc? I'm not predicting, just estimating relative importance and would like to do that on as many cases as possible.

• You can calculate variable importance for any supervised learning method, whether tree-based, neural net or regression, by applying the same method: randomly permute a predictor and see whether out-of-bag prediction quality deteriorates. The concept is simply most commonly used in Random Forests, and typically implemented out-of-the-box. For regression and similar, you would need to code it yourself. – Stephan Kolassa Feb 20 '18 at 15:50
• Tree based models do not estimate variable importance, they report an internal measument that a human decided to name "variable importance". There is nothing being estimated here in the traditional sense, so it a very different thing than a parameter estimate in regression. Be careful in overinterpreting things based on what a human decided to name them. – Matthew Drury Feb 20 '18 at 15:54

Why do we run the train-tune-test cycle in building models for prediction? Typically because different models are a priori "good" candidates, and we look at multiple ones to find the best one. (Let's not go into overfitting and biased $p$-values here.) That is the one we then use for prediction.