1
$\begingroup$

I have a dataset with multiple response variables that share the same predictor set (in particular, semantic differential scales in an attitudes task). I want to find the predictors that best explain each response variable. In a situation like this, is it valid to run random forests on each response and compare the ranks (not the magnitudes) of predictors' variable importance scores? I mention ranks because the magnitudes aren't comparable across models.

Here's an example implementation in R using party::cforest()

suppressPackageStartupMessages(library(party))
data("mammoexp", package = "TH.data")
f1 <- cforest(HIST ~ SYMPT + PB + ME + DECT, mammoexp,
              control=cforest_unbiased(ntree=50, mtry=2))
f2 <- cforest(BSE ~ SYMPT + PB + ME + DECT, mammoexp, 
              control=cforest_unbiased(ntree=50, mtry=2))
imp1 <- varimp(f1, conditional=TRUE)
imp2 <- varimp(f2, conditional=TRUE)
names(sort(imp1, decreasing=TRUE))
#> [1] "SYMPT" "PB"    "ME"    "DECT"
names(sort(imp2, decreasing=TRUE))
#> [1] "DECT"  "PB"    "ME"    "SYMPT"

Created on 2022-05-09 by the reprex package (v2.0.1)

$\endgroup$
2
  • 1
    $\begingroup$ The validity of the comparison does likely depend on the purpose of the comparison. What is the goal? To find the predictor that predicts the best across all responses? $\endgroup$
    – jluchman
    May 12 at 13:25
  • $\begingroup$ It's not finding a single best predictor, more like "which predictor predicts the best for each response?" and "what is the ranking of Predictor X for each response?" $\endgroup$ May 16 at 17:05

1 Answer 1

1
$\begingroup$

I suggest thinking about this problem like an experiment. What is being manipulated/changed and what is being controlled? Thinking about the set of models like this, you might be able to see where there is validity in making a comparison.

It sounds like the model is being run on the same data, with the same predictive model (might even recommend the use of the same random seed for all party::cforest runs), and the same predictors. These factors are (effectively) controlled for in this situation and are constant. The manipulated factors are the responses across models.

As is noted, each set of importance scores may be scaled differently by party::varimp() given they are based on different responses, however the ranks of those scores still reflect the relative rank order in which they predict each response and, as such, can be compared/contrasted as they are "normalized" within a response. Thus, the ranks allow for a comparable metric across models but is limited to their ordinal predictive usefulness.

In the end, the comparisons here do seem to allow for a valid comparison of the ordinal predictive usefulness of each of a fixed set of predictors across responses holding other factors constant.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.