0
$\begingroup$

I have a question relating to the “randomForest” package in R. I am trying to build a model with ecological variables that best explain my species occupancy data for 41 sites in the field (which I have gathered from camera traps). My ultimate goal is to do species occupancy modeling using the “unmarked” package but before I get to that stage I need to select the variables that are best explaining my occupancy, since I have many. To gain some understanding of the randomForest package I generated a fake occupancy dataset and a fake variable dataset (with variables A and D being good predictors of my occupancy and B and C being bad predictors). When I run the randomForest my output looks like this:

           0        1 MeanDecreaseAccuracy MeanDecreaseGini
A 25.3537667 27.75533           26.9634018       20.6505920
B  0.9567857  0.00000            0.9665287        0.0728273
C  0.4261638  0.00000            0.4242409        0.1411643
D 32.1889374 35.52439           34.0485837       27.0691574

        OOB estimate of  error rate: 29.02%
Confusion matrix:
    0   1 class.error
0 250 119   0.3224932
1   0  41   0.0000000

I did not make a separate train and test set, I put extra weight on the model to correctly predict the “1’s” and the variables are scaled.

I understand that this output tells me that A and D are important variables because they have high MeanDecreaseAccuracy values. However, D is the inverse of A (they are perfectly correlated) so why does D have a higher MeanDecreaseAccuracy value?

Moreover, when I run the randomForest with only A and D as variables, these values change while the confusion matrix stays the same:

         0        1 MeanDecreaseAccuracy MeanDecreaseGini
A 28.79540 29.77911             29.00879         23.58469
D 29.75068 30.79498             29.97520         24.53415

        OOB estimate of  error rate: 29.02%
Confusion matrix:
    0   1 class.error
0 250 119   0.3224932
1   0  41   0.0000000

When I run the model with only 1 good predictor (A or D) or with a good and bad predictor (AB or CD) the confusion matrix stays the same but the MeanDecreaseAccuracy values of my predictors change. Why do these values change and how should I approach the selection of my variables? (I am a beginner in occupancy modeling).

Thanks a lot!

Edit: My “real” dataset contains a lot of variables that are to some degree correlated as well. I have tried running a randomForest with this real dataset and I am confused by the results. Let me break it down: (1) In the first run I added all my variables (10), the output looked like this:

               Type of random forest: classification
                     Number of trees: 1000
No. of variables tried at each split: 3

        OOB estimate of  error rate: 29.27%
Confusion matrix:
    0   1 class.error
0 250 119  0.32249322
1   1  40  0.02439024

                              0          1 MeanDecreaseAccuracy MeanDecreaseGini
X..Primary.Forest    -5.4885443 14.9333208           -0.6418014        4.1295970
X..Secondary.Forest   3.4465544 29.5655851           14.5842266        6.8064095
Total.Forest..        2.0251384 23.7425304            9.0842793       11.5917621
X..Coffee           -11.7478635 21.5845476           -6.0780757        4.8501447
X..Grassland          3.6971609 18.5075989            9.4233284        8.3805050
X..Urban             -3.1598060 16.7651616            2.3859383        2.5009105
X..Palm               3.1110965  7.5415571            5.8375999        1.2058998
X..Mangrove          -3.0286271  0.3844779           -2.8095475        0.1073279
X..Wetlands           0.6155547 12.5150566            5.0919216        1.1475603
X..Teak              -6.8264555  6.5800623           -5.5798634        0.4720178
X..Converted          1.8151241 21.6853115            8.7168420       10.0051502

I then selected the top two variables that were not correlated, X..Secondary.Forest and X..Grassland.

(2) I ran a randomForest using just these two variables, but the model performance stays exactly the same:

               Type of random forest: classification
                     Number of trees: 1000
No. of variables tried at each split: 1

        OOB estimate of  error rate: 29.27%
Confusion matrix:
    0   1 class.error
0 250 119  0.32249322
1   1  40  0.02439024

(3) Interestingly, when I pick two variables that had less explanatory power (and were not strongly correlated), e.g. X..Converted and X..Palm, model performance is still the same (OOB = 29.27%).

What does that mean? If they all have equal explanatory properties, how do I select the variables for my model?

Thanks again!

$\endgroup$
  • $\begingroup$ Hello Fleur, welcome to CV. May I ask you what value did you use for the mtry parameter? $\endgroup$ – Davide ND Jan 23 at 12:58
  • $\begingroup$ I didn't specify the mtry parameter, so it's at the default state $\endgroup$ – Fleur Jan 23 at 14:23
0
$\begingroup$

I understand that this output tells me that A and D are important variables because they have high MeanDecreaseAccuracy values. However, D is the inverse of A (they are perfectly correlated) so why does D have a higher MeanDecreaseAccuracy value?

In general, it is not a good idea to have two perfectly correlated variables in your model. In Random Forest this simply means that you might be including the same information multiple times - in this particular case it does not cause any problem because the variable you include is the only one that brings some information, but it might be detrimental to the algorithm performance in many other cases. More importantly, having correlated variables tends to mess up your Improtances.

Why does D have higher Mean Decrease Accuracy if they are the same variable in the eyes of the model? I'd say chance. Mean Decrease Accuracy is computed making the difference of the Out Of Bag score of your model and the Out Of Bag score of the predictions of your model with a given variable permutated. Now, RF has two sources of randomness: the bootstrap, and the undersampling of the features (in your case with 4 variables and default mtry, each split is made with 2 variables), so the results could differ based on which data is bootstrapped and which features are selected in the trees. This might have particular effect if you don't have too many trees. When increasing the number of trees, the importance of the two features should converge.

A similar reasoning applies to the second question. In particular I am not sure if the variable importances are computed averaging over multiple permutations, which would be the best way to go. Indeed, since your model has 71% OOB accuracy, any accuracy drop over 21% is rather meaningless, as it means that your model goes below 50% accuracy which is random guessing.

At this link you can find some useful information on RF importances and some code for some proper Permutation Importances.

how should I approach the selection of my variables?

In this particular case you have very few variables, and it also looks like two of them are not meaningful. This means that all information is coming from just 1 variable, that you duplicated, so there is very little feature selection to be done.

In general, try to avoid highly rank correlated variables (try Spearman or Kendall correlations). Unimportant variables usually do not affect performance too much, unless there is too many of them. As a general practice though, remove useless ones from your models.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.