Valid to compare variable importance ranks across RF with different responses?

Question

I have a dataset with multiple response variables that share the same predictor set (in particular, semantic differential scales in an attitudes task). I want to find the predictors that best explain each response variable. In a situation like this, is it valid to run random forests on each response and compare the ranks (not the magnitudes) of predictors' variable importance scores? I mention ranks because the magnitudes aren't comparable across models.

Here's an example implementation in R using party::cforest()

suppressPackageStartupMessages(library(party))
data("mammoexp", package = "TH.data")
f1 <- cforest(HIST ~ SYMPT + PB + ME + DECT, mammoexp,
              control=cforest_unbiased(ntree=50, mtry=2))
f2 <- cforest(BSE ~ SYMPT + PB + ME + DECT, mammoexp, 
              control=cforest_unbiased(ntree=50, mtry=2))
imp1 <- varimp(f1, conditional=TRUE)
imp2 <- varimp(f2, conditional=TRUE)
names(sort(imp1, decreasing=TRUE))
#> [1] "SYMPT" "PB"    "ME"    "DECT"
names(sort(imp2, decreasing=TRUE))
#> [1] "DECT"  "PB"    "ME"    "SYMPT"

^{Created on 2022-05-09 by the reprex package (v2.0.1)}

Answer 1

I suggest thinking about this problem like an experiment. What is being manipulated/changed and what is being controlled? Thinking about the set of models like this, you might be able to see where there is validity in making a comparison.

It sounds like the model is being run on the same data, with the same predictive model (might even recommend the use of the same random seed for all party::cforest runs), and the same predictors. These factors are (effectively) controlled for in this situation and are constant. The manipulated factors are the responses across models.

As is noted, each set of importance scores may be scaled differently by party::varimp() given they are based on different responses, however the ranks of those scores still reflect the relative rank order in which they predict each response and, as such, can be compared/contrasted as they are "normalized" within a response. Thus, the ranks allow for a comparable metric across models but is limited to their ordinal predictive usefulness.

In the end, the comparisons here do seem to allow for a valid comparison of the ordinal predictive usefulness of each of a fixed set of predictors across responses holding other factors constant.