Just for some context, I'm asking this question from a biostastics perspective. Say you have gene expression data from multiple samples that fall under class A or class B. You then choose to build a random forest model that would try to predict whether a new sample falls in class A or class B based on gene expression. In this example, the expression level of each gene (of which there may be hundreds or thousands) would be a variable, while class A vs. B is the outcome.

After the model is built, you could then assess the relative importance of each variable (i.e. gene) in determining the classification of new data using the decrease in accuracy or Gini. You could then use these importance values to try to determine genes most associated with class A or class B. For example, you may find that the expression level of gene X is the most important variable in the model and it's expression is associated with pushing a predication towards class B. You might then conclude that gene X is associated with class B and worth further study if you interested in whatever class B is.

So far this all seems fairly straightforward and I can find many resources explaining this type of analytical approach. However, my question is how reliable are these "important" variables as the overall model accuracy decreases? If you design a model that predicts a class no better than random chance, are the "important" variables meaningful at all?

The motivation for this question is that I often see studies that publish variable importance plots without any mention of the accuracy of the associated model. These studies then go on to make claims about the importance of certain variables (in the above example it would be the importance of certain genes), solely based on the variable importance plots. If the identification of important variables is reliable with high model accuracy but not reliable at low model accuracy, this would see like a pretty large omission.

  • $\begingroup$ importance is vague term. usually in this setting it refers to the importance of the variable in the model / specifically the models predictive performance - without making claims beyond the model. $\endgroup$ – charles Dec 15 '15 at 8:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.