3
$\begingroup$

I'm trying to fit a RF model to predict tree injury categories. The data has 97,756 observations and 51 variables for which think I have enough data to predict each category:

> print(table(y)) # Check how many obs. are in each category
  y
  Grave   Leve   Muerto 
  8473    87923  1360

Using R, I believe I've selected the optimum number of predictors sampled (mtry=16) with the tuneRF() function and the number of trees with the do.trace parameter in the randomForest package function, resulting in around 200 trees. I've run the RF for a training dataset with 78,204 obs.

In the next matrix I've tested different random seeds and parameters but I cant quite figure why it's so bad the CV confusion matrix as also I don't seem to find why the RF can't predict accurately the "Muertos" category as shown in the OOB Confusion matrix.

>   set.seed(4)
>   inicio <- Sys.time()
>   ranfor <- randomForest(y ~ .,
+                          xtest=testData[,2:ncol(testData)],
+                          ytest=testData$y,
+                          data=trainData,
+                          importance=TRUE,
+                          replace=TRUE,
+                          mtry = 16,     # number of predictors sampled randomly
+                          #do.trace=50,  # Best way to trace testing RF parameters
+                          ntree=200)
>   ranfor

Call:
 randomForest(formula = y ~ ., data = trainData, xtest = testData[, 2:ncol(testData)], ytest = testData$y, importance = TRUE, replace = TRUE, mtry = 6, ntree = 200) 
               Type of random forest: classification
                     Number of trees: 200
No. of variables tried at each split: 16

        OOB estimate of  error rate: 3%
Confusion matrix:
       Grave  Leve Muerto  class.error
Grave   5556  1230     18 0.1834215168
Leve      23 70291      0 0.0003271041
Muerto   971   101     14 0.9871086556
                Test set error rate: 2.94%
Confusion matrix:
       Grave  Leve Muerto  class.error
Grave   1370   299      0 0.1791491911
Leve       4 17605      0 0.0002271566
Muerto   253    19      2 0.9927007299
>   (duracion <- Sys.time()-inicio) # Check computational timing
Time difference of 8.169226 mins

I'm really not satisfied with the OOB overall estimate error rate as most of the categories are "Leve" (90% of obs.) and I'm not sure if I should weight the categories (don't know how) so that "Leve" gets less importance than the other two. As "Muertos" or "Grave" are more important categories to predict, those are also the ones with relatively fewer obs.

It seems (in my opinion) that many "Graves" and "Muertos" are getting misclassified into a worst scenario ("Leve" is the likeliest injury , "grave" means severe injury, while"muertos" means dead) as 253 "Graves" are misclassified as "Muertos", and the same happens for 299 "Leves".

Edited: I also found out (something strange for me) that by seeing the variable importance, one explanatory variable which is VISIBILIDAD_RESTRINGIDA(restricted visibility, let's just say "ResVis") is way more important than the others. Let me clarify that MOST variables are categorical, even this one, and I specified as.factor() before running the RF.

> table(external$VISIBILIDAD_RESTRINGIDA,y)
     y
      Grave  Leve Muerto
  1    5052  3194    896
  2     135    41      7
  3     545   178    167
  4     149   165     32
  5     188    22     12
  6      78     8      8
  7     358   812     48
  999  1968 83503    190

> importance(ranfor)
                               Grave        Leve       Muerto MeanDecreaseAccuracy MeanDecreaseGini
MES                        2.5499660   1.1561400  -1.78931277            1.9598768      273.4210874
HORA                      -0.4439739   8.5604395   2.46446030            8.2100764      334.6225009
DIASEMANA                  1.0019474   5.8850148   3.02968724            5.9603441      316.7660137
PROVINCIA                 13.1010470  19.3139535   0.33016912           21.4222718      879.2420003
COMUNIDAD_AUTONOMA        18.2752637  28.2041723   2.94547590           30.9687114     1559.0596558
TOT_VEHICULOS_IMPLICADOS   5.6190772  14.2704791  -0.87260074           14.1312291       83.4322001
ZONA                       6.9706570   4.5873052   2.33176751            6.6401112       31.4029963
ZONA_AGRUPADA              6.6862350   4.5912797  -3.81985526            6.7918270       14.3216491
RED_CARRETERA              4.1287431  11.7429400   5.87706042           13.3615428      119.1686018
TIPO_VIA                   9.2367252   6.8730783   1.92847612           10.2634635      122.0410134
TRAZADO_NO_INTERSEC        9.7449677  12.8801960  -0.01817554           16.3323278       97.8624425
TIPO_INTERSEC              3.8144943  14.7940975  -3.26975823           15.1859458       98.5696311
PRIORIDAD_AGENTE           0.0000000  -2.0198301   0.00000000           -2.0229832        0.8308287
PRIORIDAD_SEMAFORO        -2.8899188   4.3570527  -1.30401691            4.1355284        6.7000675
PRIORIDAD_STOP            -4.1117093   7.7285086   2.12207132            5.0658263       11.1273896
PRIORIDAD_CEDA            -0.8340772   2.9711414  -3.28403522            2.8715671        8.2193895
PRIORIDAD_MARCAS          -0.9390412   1.6881347  -0.06374335            0.6483825        2.9137353
PRIORIDAD_PASO             1.5027717   0.3546953   1.42203645            1.1666489        5.1521207
PRIORIDAD_OTRA             0.1920776   2.7565216   1.45609080            2.8494874        6.6007651
SUPERFICIE_CALZADA         2.1470855  13.4549320  -3.27276700           13.5332254       43.6284663
LUMINOSIDAD               -1.5990166  14.0844006   7.25078475           13.9417434       91.2584278
FACTORES_ATMOSFERICOS      6.2550478  12.5924679  -0.42075889           14.3637256       83.9186203
VISIBILIDAD_RESTRINGIDA  448.9797516 133.0524840 106.03085701          194.8708903     5478.4219795
ACERAS                     8.9853725   8.4241373   7.20036912           10.6043738       64.1523934
TIPO_ACCIDENTE            10.1565640  29.1024414   5.40672767           31.3913280      341.8189757
EDAD                      -0.7095702  25.2851389  -1.60128832           25.1090462      384.7965295
SEXO                      -0.1310307   4.6014676   1.07798972            4.4432653       47.1488975
ANIO_PERMISO              -1.5286530  21.5785372   0.69529565           21.9758923      837.4107857
POSICION                   3.2640438   8.8990406  -3.63864409            8.9721442       19.2839216
USO_CINTURON               6.0512216  16.2972098   2.89675498           17.7820318       71.3990573
USO_SRI                   -1.4212803   1.0025094   0.00000000           -0.5785161        0.1514192
USO_CASCO                  5.7632639   8.8483903  -1.54018878           10.2937100       44.4668913
MANIOBRAS                  6.1361490   9.4391468  -6.13622536           10.8650779      228.1015341
INFRACC_VELOCIDAD         10.9436343   8.8493527   7.78231571           10.8850554      161.5953998
INFRACC_COND              13.7179808  11.5304944  -0.75954497           14.4900487      399.5694907
INFRACC_APERTURA           6.0522070   4.4034781   0.71227196            6.1263177       52.0423852
INFRACC_ALUMBRADO          7.3306091   6.2272118  -1.19265165            8.8935766       44.9046637
INFRACC_CARGA_VEHICULO     5.0378570   6.6349341   2.20374400            7.1518982       84.5196677
INFRACC_RESUMEN            8.0811364   7.3897426   1.45865212            9.4172296      171.9358202
INFRACC_PEATON            -2.8033339   0.9499664   1.67683121            0.1851498        0.9939418
ANIO_MATRICULA_VEHICULO    5.9295547  11.4859615   4.35343730           11.7072978      365.0703755
MES_MATRICULA_VEHICULO     3.4049466  15.7972570  -0.53687395           16.0690088      499.7145951
TIPO_VEHICULO              6.6588133  21.2787095  -0.19610130           20.6711130      199.0782706
ANOMALIA_NINGUNA           9.3328101   6.8778253   5.88453633            8.2228360      143.6617389
ANOMALIA_NEUMATICO         7.8485532   6.8721105   6.70488013            7.4067129      119.1310958
ANOMALIA_REVENTON          6.5487401   5.8373472   4.10660551            6.1963343       74.3142386
ANOMALIA_DIRECCION         5.4273533   4.5115241   3.42907781            4.7797505       66.9991264
ANOMALIA_FRENOS            7.0495934   7.1336800   5.96273787            7.3882427      102.2889357
NUMERO_OCUPANTES_VEH       2.0068816  17.4207823  -2.45100714           17.7556118      140.4688852
MERCANCIAS_PELIGROSAS     -5.0237694  -0.6334626   2.38000946           -4.5311280        1.0890107
VEHICULO_INCENDIADO       -3.5096342  -0.8431701   4.65061584           -0.4692923        5.3178919

I'd really appreciate any help, comment, or suggestion about the matter at hand or maybe correction about my lack of expression of the problem. Thanks in advance!

Improve this question
$\endgroup$
  • 1
    $\begingroup$ A comment, because this isn't a full treatment of your question: in the rpart package, you can explicitly define a loss matrix which allows you to give a larger penalty to, say, misclassifying "Death" as "Light Injury". It looks like the randomForest package doesn't have builtin support for this, at least with a quick search. You might find some useful discussion here. And if you do enough Googling maybe you'll find someone has implemented it and posted their code online. $\endgroup$ – klumbard May 17 '18 at 13:34
  • 1
    $\begingroup$ might consider the "h2o" package implementation of random forest. It is fast, and has more handrails, reporting, and pretty plots. $\endgroup$ – EngrStudent May 17 '18 at 19:18
1
$\begingroup$

The problem is you have a very unbalanced data set, that is class "Leve" is far more frequent then then other two classes. This inevitably results in your RF classifier being highly biased towards the majority class and so the majority of your cases will be classified in the majority class, as that will likely maximize the Error rate (Misclassification rate).

To avoid this issue you need to use a different metric from the Error Rate, for example

  • Precision
  • Recall
  • F-score
  • etc.

Wikipedia has lots of information on the various metric that can be calculated from a confusion matrix.

Alternatively you could give different weight to the classes using the argument classwt, however this is approach is not trivial.

Improve this answer
$\endgroup$
  • $\begingroup$ Indeed, ‘Leve’ is way more frequent than the others. I’ve tried allocating weights to the RF but because of my lack of coding skills, I couldn’t manage to do it (maybe in a near future). So I separated ‘Muertos’ from the sample and fitted a binary GLM. The results improved a lot, now the RF model had a 96% accuracy to determine the car crash injury level, and the GLM could predict in 98% of the cases if the crash was fatal or not. Thanks a lot for the support, I’ll check out those metrics for the error ;) $\endgroup$ – AR_Domingo Sep 10 '18 at 16:50
0
$\begingroup$

Forgot to reply my solution. The main reason of this output was that the variable muertos which was very noisy inside the random forest. So to avoid modifying the RF (which wasn’t an option due to time restrictions) I’ve choose to separate this variable and fitted a GLM to it.

I think the problem could’ve be solved using the Random Forest with variable weights, but I couldn’t try this approach.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.