Necessary to train, tune and test if only estimating variable importance? In medicine the use of regression models may differ slightly from other fields. We usually build regression models from theory and subject matter knowledge. We're typically interested in estimating the effect of some clinical feature, while accounting for other variables. Less often, we develop more typical prediction models and in those circumstances we do respect the fact that one must train, tune and test models by conventional means.
However, I'm looking into tree based algorithms (random forest and GBM) and was thinking about this. I'm estimating the effect of a particular treatment, and I'm interested in the relative importance of that treatment. I'm not interested in predicting per se. As compared with regression models, tree based methods seem to have an inherent ability to provide estimates of relative variable importance, which is beautiful.
Can I apply the same thinking in machine learning? Can I skip the training and test procedure, fine tuning etc? I'm not predicting, just estimating relative importance and would like to do that on as many cases as possible.
 A: Why do we run the train-tune-test cycle in building models for prediction? Typically because different models are a priori "good" candidates, and we look at multiple ones to find the best one. (Let's not go into overfitting and biased $p$-values here.) That is the one we then use for prediction.
The same logic applies if you want to assess variable importance. The importance of a variable does not live in a vacuum. In the population, it exists in the context of all the other effects on your dependent variables. And in your sample, it exists in the context of all other effects and your sample, i.e., your model.
So: if you are certain that you know the "best" model beforehand, you don't need to calibrate it. This is only very rarely the case, both for prediction and for assessing variable importance. So you will need to check multiple models to find one that describes your data well.
So there is a good case for training, tuning and testing even if you are only interested in variable importance.
And of course, you can overfit quite as easily in this context as in any other.
