In medicine the use of regression models may differ slightly from other fields. We usually build regression models from theory and subject matter knowledge. We're typically interested in estimating the effect of some clinical feature, while accounting for other variables. Less often, we develop more typical prediction models and in those circumstances we do respect the fact that one must train, tune and test models by conventional means.
However, I'm looking into tree based algorithms (random forest and GBM) and was thinking about this. I'm estimating the effect of a particular treatment, and I'm interested in the relative importance of that treatment. I'm not interested in predicting per se. As compared with regression models, tree based methods seem to have an inherent ability to provide estimates of relative variable importance, which is beautiful.
Can I apply the same thinking in machine learning? Can I skip the training and test procedure, fine tuning etc? I'm not predicting, just estimating relative importance and would like to do that on as many cases as possible.