0
$\begingroup$

I'm trying to fit a Random Forest model in R. I've got a DF with 13 factorized variables plus the target one (binary). When I try to use all the variables with

fitRf <- randomForest(fl_ok ~ .,   data=train)

The result printing the model is

    OOB estimate of  error rate: 35.35%
Confusion matrix:
     0    1 class.error
0 6492  263  0.03893412
1 4546 2303  0.66374653

The table with variables importance is

PROVINCIA_RESIDENZA_CONTRAENTE         989.2589
REGIONE                                550.9323
FASCIA_ETA                             602.3096
SESSO                                  153.0396
COD_CATEGORIA_VEICOLO                  150.4787
COD_ZONA_DANNEGGIATA                   562.0613
RUOLO                                  293.8663
TIPO_SX                                 99.2244
NUM_FIRME                              132.7890
TIPO_LESIONE                           506.2868
int_legale                             673.7938
gg_ap_den_lesioni                      260.0141
life_cycle                             445.2724

When I try to leave out the most important variable ('PROVINCIA_RESIDENZA_CONTRAENTE') the model performance is better:

OOB estimate of  error rate: 29.37%
Confusion matrix:
     0    1 class.error
0 4594 2161   0.3199112
1 1835 5014   0.2679223

Note that the left out variable is a factor with 45 levels representing the County of the single record. Why is this happening? Is there anything I can do to use this variable in the model?

Improve this question
$\endgroup$
3
  • $\begingroup$ Use cross-validation and examine out-of-sample performance. $\endgroup$
    – Sycorax ♦
    May 16, 2016 at 14:43
  • $\begingroup$ Isn't that the OOB estimate in the report above? $\endgroup$
    – Matteo Felici
    May 16, 2016 at 15:32
  • $\begingroup$ No. Those are only out-of-sample in one tree. I've observed that OOB estimates tend not to have a whole lot to do with the estimates obtained by a separate hold-out set. $\endgroup$
    – Sycorax ♦
    May 16, 2016 at 18:16

1 Answer 1

Reset to default
3
$\begingroup$

The reason could be that the categorical variable you are leaving out has too many levels compared the other variables. From The Elements of Statistical Learning:

The partitioning algorithm tends to favor categorical predictors with many levels q; the number of partitions grows exponentially in q, and the more choices we have, the more likely we can find a good one for the data at hand. This can lead to severe overfitting if q is large, and such variables should be avoided.

(at paragraph 9.2.4, page 310 -some more references included there).

Is is the case that the other 12 categorical variables have much fewer levels compared to the province/county? If that's the case, this could be your issue. From the results you are providing, you could leave it out, also taking into account that "REGIONE" must be (I guess) a higher level version of the info in the county column.

Hope it helps.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.