Random Forest in R - most important variable causing errors

Question

I'm trying to fit a Random Forest model in R. I've got a DF with 13 factorized variables plus the target one (binary). When I try to use all the variables with

fitRf <- randomForest(fl_ok ~ .,   data=train)

The result printing the model is

    OOB estimate of  error rate: 35.35%
Confusion matrix:
     0    1 class.error
0 6492  263  0.03893412
1 4546 2303  0.66374653

The table with variables importance is

PROVINCIA_RESIDENZA_CONTRAENTE         989.2589
REGIONE                                550.9323
FASCIA_ETA                             602.3096
SESSO                                  153.0396
COD_CATEGORIA_VEICOLO                  150.4787
COD_ZONA_DANNEGGIATA                   562.0613
RUOLO                                  293.8663
TIPO_SX                                 99.2244
NUM_FIRME                              132.7890
TIPO_LESIONE                           506.2868
int_legale                             673.7938
gg_ap_den_lesioni                      260.0141
life_cycle                             445.2724

When I try to leave out the most important variable ('PROVINCIA_RESIDENZA_CONTRAENTE') the model performance is better:

OOB estimate of  error rate: 29.37%
Confusion matrix:
     0    1 class.error
0 4594 2161   0.3199112
1 1835 5014   0.2679223

Note that the left out variable is a factor with 45 levels representing the County of the single record. Why is this happening? Is there anything I can do to use this variable in the model?

Answer 1

The reason could be that the categorical variable you are leaving out has too many levels compared the other variables. From The Elements of Statistical Learning:

The partitioning algorithm tends to favor categorical predictors with many levels q; the number of partitions grows exponentially in q, and the more choices we have, the more likely we can find a good one for the data at hand. This can lead to severe overfitting if q is large, and such variables should be avoided.

(at paragraph 9.2.4, page 310 -some more references included there).

Is is the case that the other 12 categorical variables have much fewer levels compared to the province/county? If that's the case, this could be your issue. From the results you are providing, you could leave it out, also taking into account that "REGIONE" must be (I guess) a higher level version of the info in the county column.

Hope it helps.