1
$\begingroup$

I am running regression random forests.

My original database has 138 potential predictors (x.1, x.2 ... x.138) and 4 different response variables (a, b, c and d). The response variables are different ways to quantify the same phenomenon (complexity in the architecture of a plant).

After carrying out a feature selection for every response variable, I finished with 19 predictors. Some predictors were kept exclusively on one model, while others were kept in more than one model:


RF_a (selected features): x.1, x.2, x.3, x.4, x.5, x.6, x.7, x.8, x.9
RF_b (selected features): x.1, x.6, x.7, x.8, x.10, x.11, x.12
RF_c (selected features): x.3, x.7, x.8, x.9, x.13, x.14, x.15, x.16, x.17, x.18
RF_d (selected features): x.6, x.7, x.8, x.9, x.19


So, for example, predictor x.6 is common to models RF_a, RF_b and RF_d, while predictor x.19 was kept only for model RF_d.

For each of the final models (RF_a, RF_b, RF_c, RF_d) I have the % of variance explained by the model, the mean of squared residuals, and I can also compute the importance of each predictor variable as explained here. These Importance measures of the predictors are specific to each model and indicate the differences (in percentaje) generated in the MSE of the model after permuting each predictor variable.

I am trying to generate a ranking of the Importance of the variables throughout all the models, to get an idea of which variables were the most relevant throughout the modeling process. Averaging the "variable importance" values of a predictor among all the models in which that predictor appears makes no sense since a predictor can be the more important in a group of poor models. So I wonder if there is a way to weigh the "variable Importance" by some measure of the overall performance of each model.

Some data and the models! https://drive.google.com/file/d/1lH-qn14EHHTrtqm5-4TB7be2HZGDvN2R/view?usp=sharing

    library("randomForest")
    ## ##########################
    ## ##########################
    Data <- as.data.frame(
        data.table::fread("For_stackoverflow.txt"))

    ## ##########################
    y <- Data[, "a"]
    x <- Data[, c("x.1", "x.2", "x.3", "x.4",
                  "x.5", "x.6", "x.7", "x.8",
                  "x.9")]
    Fita <- randomForest(x = x, y = y,
                         mtry = 3,
                         ntree = 20000,
                         importance = T)
    Fita
    importance(Fita, type = 1)

    ## ##########################
    y <- Data[, "b"]
    x <- Data[, c("x.1", "x.6", "x.7", "x.8",
                  "x.10", "x.11", "x.12")]
    Fitb <- randomForest(x = x, y = y,
                         mtry = 3,
                         ntree = 20000,
                         importance = T)
    Fitb
    importance(Fitb, type = 1)

    ## ##########################
    y <- Data[, "c"]
    x <- Data[, c("x.3", "x.7", "x.8", "x.9",
                  "x.13", "x.14", "x.15", "x.16",
                  "x.17","x.18")]
    Fitc <- randomForest(x = x, y = y,
                         mtry = 3,
                         ntree = 20000,
                         importance = T)
    Fitc
    importance(Fitc, type = 1)

    ## ##########################
    y <- Data[, "d"]
    x <- Data[, c("x.6", "x.7", "x.8",
                  "x.9", "x.19")]
    Fitd <- randomForest(x = x, y = y,
                         mtry = 2,
                         ntree = 20000,
                         importance = T)
    Fitd
    importance(Fitd, type = 1)
$\endgroup$
  • $\begingroup$ Fun problem! (+1) I think your intuition that the naive mean is misleading is absolute correct; please see my post below for more details and a sketch of a solution. $\endgroup$ – usεr11852 Jan 25 at 0:31
0
$\begingroup$

I think it would be more relevant to use a dimensionality reduction step first and then uses the resulting variables as the new responses.

Assuming one uses Principal Component Analysis (I mention it as it is the simplest and most widely used dimensionality reduction technique), this would be akin to do Principal Component Regression and then looking at the variable importance within each orthogonal mode of variation. For example, if the principal component (PC) scores $\xi_1$ from the first PC are encapsulating a large proportion of the variation observed among the four metrics used, the most important feature (e.g. $x_6$) explaining a substantial portion of the variation in $\xi_1$ itself, would indeed be one of the most important features.

Notice that PCA in itself is just a linear combination of our original data such that the resulting covariance matrix is diagonal (i.e. the resulting PC scores $\Xi$ are orthogonal to each other). If through expert knowledge you know that for example one of the complexity metrics is more important, it is perfectly reasonable to weight it accordingly. You are correct question the validity of taking the naive mean of feature importances when using the raw response variables for modelling. To that extent, we can go ahead and take the weighted average of the feature importance scores across these four RF modelling the PC scores; weighting them by the percentage of variance explain from their corresponding PC. This will give a more accurate view of what are the most important features across all four metrics as it will encapsulate the variance explained along orthogonal modes of variation scaled by their respective variance magnitude (the eigenvalues $\lambda$).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.