In medicine the use of regression models may differ slightly from other fields. We usually build regression models from theory and subject matter knowledge. We're typically interested in estimating the effect of some clinical feature, while accounting for other variables. Less often, we develop more typical prediction models and in those circumstances we do respect the fact that one must train, tune and test models by conventional means.

However, I'm looking into tree based algorithms (random forest and GBM) and was thinking about this. I'm estimating the effect of a particular treatment, and I'm interested in the relative importance of that treatment. I'm not interested in predicting per se. As compared with regression models, tree based methods seem to have an inherent ability to provide estimates of relative variable importance, which is beautiful.

Can I apply the same thinking in machine learning? Can I skip the training and test procedure, fine tuning etc? I'm not predicting, just estimating relative importance and would like to do that on as many cases as possible.

  • 1
    $\begingroup$ You can calculate variable importance for any supervised learning method, whether tree-based, neural net or regression, by applying the same method: randomly permute a predictor and see whether out-of-bag prediction quality deteriorates. The concept is simply most commonly used in Random Forests, and typically implemented out-of-the-box. For regression and similar, you would need to code it yourself. $\endgroup$ – Stephan Kolassa Feb 20 '18 at 15:50
  • $\begingroup$ Tree based models do not estimate variable importance, they report an internal measument that a human decided to name "variable importance". There is nothing being estimated here in the traditional sense, so it a very different thing than a parameter estimate in regression. Be careful in overinterpreting things based on what a human decided to name them. $\endgroup$ – Matthew Drury Feb 20 '18 at 15:54

Why do we run the train-tune-test cycle in building models for prediction? Typically because different models are a priori "good" candidates, and we look at multiple ones to find the best one. (Let's not go into overfitting and biased $p$-values here.) That is the one we then use for prediction.

The same logic applies if you want to assess variable importance. The importance of a variable does not live in a vacuum. In the population, it exists in the context of all the other effects on your dependent variables. And in your sample, it exists in the context of all other effects and your sample, i.e., your model.

So: if you are certain that you know the "best" model beforehand, you don't need to calibrate it. This is only very rarely the case, both for prediction and for assessing variable importance. So you will need to check multiple models to find one that describes your data well.

So there is a good case for training, tuning and testing even if you are only interested in variable importance.

And of course, you can overfit quite as easily in this context as in any other.

  • $\begingroup$ Thanks Stephan. I agree with you that a good model is better in the way you describe. However, that's not what they do in medical research (I read these papers daily) and some prominent statisticians in the field (e.g Frank Harrell) also suggest that for effect estimation one can build models from subject matter knowledge (refer to Regression Modelling Strategies, Ch 1, summary). Now that I started doing a bit of testing-tuning-training on the data, there are no noteworthy differences in final results. With that said, I still get your point and realize it is a wise approach. $\endgroup$ – Jennifer Mente Feb 20 '18 at 16:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.