Can I safely use variable importance of a random forest in a paper?

Question

Background: I just started with machine learning and I'm considering using it on old data based on which I'm writing a paper. The paper deals with radiation-induced lung damage and the data comprise breathing rate measurements, as well as different types of histological scorings for each animal.

One of the questions I would like to answer in the paper is if the histology is a predictor of the lung damage, and if yes, which feature of the histology is the most important predictor. The outcome "lung damage" is a boolean value based on the breathing frequency.

My idea was to let a random forest predict the outcome using the scoring data and report the important variables in the paper with the statement that "Scorings of the histological features X, Y and Z are the best predictors for lung damage". IMO using machine learning to do that would give me a qualitative measure of important variables, but would spare me the task of developing a complicated model myself which predicts the outcome, as this is not the main point of the paper.

My questions are:

1. Is this possible and a good idea?
2. Is the variable importance a robust measure, or will slight changes in the data lead to an entirely different variable importance ranking?
3. Is the accuracy of the prediction important in this context and how accurate must the prediction be in order for me to be sure that the variable importance is right?

Thank you very much for your insights!

Answer 1

It does not really seem that you can justify the "X, Y and Z are best predictors" sentence in this case. At least because all predictors are best for purpose, i.e. are they so specific that they can be used as the final truth in diagnosis, or are they so sensitive that given some predictor values, no cases will be missed, or maybe those perform better than others on average?

What you can state is exactly what you obtained: X, Y and Z scored best on the scale of variable importance of the RandomForest algorithm.

It looks that you studied the association of various predictors with the outcome, a type of study that many researchers do, so I would encourage you to use the de facto standard of reporting association in medical and biological research, namely the combination of odds ratio (effect size) and the p-value for exact Fisher's test. Such measures are reported very often (if not always) and allow other researchers to compare results between papers.

Of course the importance metric will not hurt anyone if you add it to the most commonly used two.

Answer 2

Would agree with everything coulminer answered above. Would add a few points that I'm not sure are useful:
- It might be hard to justify using RF over more traditional methods. You'd likely need to emphasize a combination of large number of variables + unknown number of interactions + non-linear effects to be convincing.
- The importance measure try to measure the importance within the RF model. Nothing more. Unless you're building the RF model for other reasons they likely won't add anything. gbm will produce different measures of variable importance - again specific to that model.
- Boruta and similar packages try to find a all-relevant subset of features. I would use them for this purpose, and not put too much weight on the variable importance they produce.
- There is more than one RF variable importance measure. There are two in the randomForest package. One in the party package. A separate one in the randomForestSRC package. You could even use the rminer package. Variable importance may change if you change the sampling (internal undersampling) --> don't think there is a perfect variable importance measure, or one you should put too much faith into.