0
$\begingroup$

I'm trying to determine variable importance for a random forest with 8 predictors, and different variable importance measures are telling different stories. The forest was generated in R with randomForest (with localImp=TRUE), and the measures come from randomForestExplainer's measure_importance (with the default mean_sample="top_trees").

> fstLH <- randomForest(Blackness ~ ., 
                        resultsWithProsod[resultsWithProsod$Accent=="LH", !(colnames(resultsWithProsod) %in% ignoreCols)],
                        localImp=TRUE)
> impFrmLH <- measure_importance(fstLH)
> c(lapply(impFrmLH[2], function(x) {
      setNames(sort(x, decreasing=F), impFrmLH$variable[order(x, decreasing=F)])
    }),
    lapply(impFrmLH[-(1:2)], function(x) {
      setNames(sort(x, decreasing=T), impFrmLH$variable[order(x, decreasing=T)])
  }))
$mean_min_depth
PhraseSpeechRate        PeakDelay              HNR     IntensityAvg 
        2.003904         2.007264         2.116000         2.119136 
   F0MaxMinRatio          Shimmer           Jitter             Step 
        2.185136         2.355136         2.366000         3.093680 

$no_of_nodes
   F0MaxMinRatio     IntensityAvg              HNR           Jitter 
            2866             2837             2827             2762 
         Shimmer             Step PhraseSpeechRate        PeakDelay 
            2684             2198             1486             1453 

$mse_increase
PhraseSpeechRate        PeakDelay              HNR     IntensityAvg 
     0.091133749      0.080856088      0.048558841      0.048251741 
   F0MaxMinRatio          Shimmer           Jitter             Step 
     0.038783659      0.018955181      0.015282364      0.004449853 

$node_purity_increase
PhraseSpeechRate        PeakDelay     IntensityAvg              HNR 
        97.10995         79.07907         54.43092         49.60133 
   F0MaxMinRatio          Shimmer           Jitter             Step 
        48.69803         30.74844         28.62819         11.86899 

$no_of_trees
             HNR           Jitter    F0MaxMinRatio     IntensityAvg 
             500              500              499              499 
         Shimmer             Step PhraseSpeechRate        PeakDelay 
             499              495              486              476 

$times_a_root
       PeakDelay PhraseSpeechRate              HNR    F0MaxMinRatio 
             114              103               79               57 
          Jitter          Shimmer     IntensityAvg             Step 
              54               47               46                0 

$p_value
       PeakDelay PhraseSpeechRate             Step          Shimmer 
    1.000000e+00     1.000000e+00     9.999887e-01     1.198402e-10 
          Jitter              HNR     IntensityAvg    F0MaxMinRatio 
    7.488331e-16     4.964456e-21     6.792990e-22     1.685192e-24 

According to mean minimal depth, MSE increase, node purity increase, and times-a-root, two predictors (PhraseSpeechRate and PeakDelay) are clearly most important to the forest. But these two predictors also show up in the fewest nodes and fewest trees, and they get a (suspicious) p-value of 1.

Is there some interaction I'm missing here, or do I need to adjust tuning parameters of the random forest?

$\endgroup$
  • $\begingroup$ They all measure different things, so there's no underlying reason they would agree. $\endgroup$ – Matthew Drury Aug 3 '18 at 2:56
  • $\begingroup$ Right, I'm not saying they should all be in lockstep, but certain disagreements are curious. Intuitively, I would expect that if a variable is likelier than any other to be a root node, then it would also likely show up frequently as a non-root node; but PhraseSpeechRate and PeakDelay are the commonest root nodes but least common (root or non-root) nodes. $\endgroup$ – Dan Villarreal Aug 3 '18 at 3:50
  • $\begingroup$ Perhaps a better way to frame the question would've been this: under what conditions would we expect these variable importance measures to (strongly) disagree in this way? $\endgroup$ – Dan Villarreal Aug 3 '18 at 3:58
  • 1
    $\begingroup$ It's hard to tell without having the complete code. However, the p-value does not indicate which variables are most important. An informative variable would probably split near the root node, so would be less likely to split further down (still possible) but have small minimal depth. You can see in the following vignette that sometimes variables with relatively smaller minimal depth have larger p-values: cran.rstudio.com/web/packages/randomForestExplainer/vignettes/… $\endgroup$ – Peter Calhoun Aug 5 '18 at 6:36
  • $\begingroup$ I see. I thought the high rate of selection as a root node might trigger a lower rate of splitting further down the tree. And I've added the rest of the code. $\endgroup$ – Dan Villarreal Aug 5 '18 at 23:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.