I'm trying to fit a RF model to predict tree injury categories. The data has 97,756 observations and 51 variables for which think I have enough data to predict each category:
> print(table(y)) # Check how many obs. are in each category
y
Grave Leve Muerto
8473 87923 1360
Using R, I believe I've selected the optimum number of predictors sampled (mtry
=16) with the tuneRF()
function and the number of trees with the do.trace
parameter in the randomForest
package function, resulting in around 200 trees. I've run the RF for a training dataset with 78,204 obs.
In the next matrix I've tested different random seeds and parameters but I cant quite figure why it's so bad the CV confusion matrix as also I don't seem to find why the RF can't predict accurately the "Muertos" category as shown in the OOB Confusion matrix.
> set.seed(4)
> inicio <- Sys.time()
> ranfor <- randomForest(y ~ .,
+ xtest=testData[,2:ncol(testData)],
+ ytest=testData$y,
+ data=trainData,
+ importance=TRUE,
+ replace=TRUE,
+ mtry = 16, # number of predictors sampled randomly
+ #do.trace=50, # Best way to trace testing RF parameters
+ ntree=200)
> ranfor
Call:
randomForest(formula = y ~ ., data = trainData, xtest = testData[, 2:ncol(testData)], ytest = testData$y, importance = TRUE, replace = TRUE, mtry = 6, ntree = 200)
Type of random forest: classification
Number of trees: 200
No. of variables tried at each split: 16
OOB estimate of error rate: 3%
Confusion matrix:
Grave Leve Muerto class.error
Grave 5556 1230 18 0.1834215168
Leve 23 70291 0 0.0003271041
Muerto 971 101 14 0.9871086556
Test set error rate: 2.94%
Confusion matrix:
Grave Leve Muerto class.error
Grave 1370 299 0 0.1791491911
Leve 4 17605 0 0.0002271566
Muerto 253 19 2 0.9927007299
> (duracion <- Sys.time()-inicio) # Check computational timing
Time difference of 8.169226 mins
I'm really not satisfied with the OOB overall estimate error rate as most of the categories are "Leve" (90% of obs.) and I'm not sure if I should weight the categories (don't know how) so that "Leve" gets less importance than the other two. As "Muertos" or "Grave" are more important categories to predict, those are also the ones with relatively fewer obs.
It seems (in my opinion) that many "Graves" and "Muertos" are getting misclassified into a worst scenario ("Leve" is the likeliest injury , "grave" means severe injury, while"muertos" means dead) as 253 "Graves" are misclassified as "Muertos", and the same happens for 299 "Leves".
Edited: I also found out (something strange for me) that by seeing the variable importance, one explanatory variable which is VISIBILIDAD_RESTRINGIDA
(restricted visibility, let's just say "ResVis") is way more important than the others. Let me clarify that MOST variables are categorical, even this one, and I specified as.factor()
before running the RF.
> table(external$VISIBILIDAD_RESTRINGIDA,y)
y
Grave Leve Muerto
1 5052 3194 896
2 135 41 7
3 545 178 167
4 149 165 32
5 188 22 12
6 78 8 8
7 358 812 48
999 1968 83503 190
> importance(ranfor)
Grave Leve Muerto MeanDecreaseAccuracy MeanDecreaseGini
MES 2.5499660 1.1561400 -1.78931277 1.9598768 273.4210874
HORA -0.4439739 8.5604395 2.46446030 8.2100764 334.6225009
DIASEMANA 1.0019474 5.8850148 3.02968724 5.9603441 316.7660137
PROVINCIA 13.1010470 19.3139535 0.33016912 21.4222718 879.2420003
COMUNIDAD_AUTONOMA 18.2752637 28.2041723 2.94547590 30.9687114 1559.0596558
TOT_VEHICULOS_IMPLICADOS 5.6190772 14.2704791 -0.87260074 14.1312291 83.4322001
ZONA 6.9706570 4.5873052 2.33176751 6.6401112 31.4029963
ZONA_AGRUPADA 6.6862350 4.5912797 -3.81985526 6.7918270 14.3216491
RED_CARRETERA 4.1287431 11.7429400 5.87706042 13.3615428 119.1686018
TIPO_VIA 9.2367252 6.8730783 1.92847612 10.2634635 122.0410134
TRAZADO_NO_INTERSEC 9.7449677 12.8801960 -0.01817554 16.3323278 97.8624425
TIPO_INTERSEC 3.8144943 14.7940975 -3.26975823 15.1859458 98.5696311
PRIORIDAD_AGENTE 0.0000000 -2.0198301 0.00000000 -2.0229832 0.8308287
PRIORIDAD_SEMAFORO -2.8899188 4.3570527 -1.30401691 4.1355284 6.7000675
PRIORIDAD_STOP -4.1117093 7.7285086 2.12207132 5.0658263 11.1273896
PRIORIDAD_CEDA -0.8340772 2.9711414 -3.28403522 2.8715671 8.2193895
PRIORIDAD_MARCAS -0.9390412 1.6881347 -0.06374335 0.6483825 2.9137353
PRIORIDAD_PASO 1.5027717 0.3546953 1.42203645 1.1666489 5.1521207
PRIORIDAD_OTRA 0.1920776 2.7565216 1.45609080 2.8494874 6.6007651
SUPERFICIE_CALZADA 2.1470855 13.4549320 -3.27276700 13.5332254 43.6284663
LUMINOSIDAD -1.5990166 14.0844006 7.25078475 13.9417434 91.2584278
FACTORES_ATMOSFERICOS 6.2550478 12.5924679 -0.42075889 14.3637256 83.9186203
VISIBILIDAD_RESTRINGIDA 448.9797516 133.0524840 106.03085701 194.8708903 5478.4219795
ACERAS 8.9853725 8.4241373 7.20036912 10.6043738 64.1523934
TIPO_ACCIDENTE 10.1565640 29.1024414 5.40672767 31.3913280 341.8189757
EDAD -0.7095702 25.2851389 -1.60128832 25.1090462 384.7965295
SEXO -0.1310307 4.6014676 1.07798972 4.4432653 47.1488975
ANIO_PERMISO -1.5286530 21.5785372 0.69529565 21.9758923 837.4107857
POSICION 3.2640438 8.8990406 -3.63864409 8.9721442 19.2839216
USO_CINTURON 6.0512216 16.2972098 2.89675498 17.7820318 71.3990573
USO_SRI -1.4212803 1.0025094 0.00000000 -0.5785161 0.1514192
USO_CASCO 5.7632639 8.8483903 -1.54018878 10.2937100 44.4668913
MANIOBRAS 6.1361490 9.4391468 -6.13622536 10.8650779 228.1015341
INFRACC_VELOCIDAD 10.9436343 8.8493527 7.78231571 10.8850554 161.5953998
INFRACC_COND 13.7179808 11.5304944 -0.75954497 14.4900487 399.5694907
INFRACC_APERTURA 6.0522070 4.4034781 0.71227196 6.1263177 52.0423852
INFRACC_ALUMBRADO 7.3306091 6.2272118 -1.19265165 8.8935766 44.9046637
INFRACC_CARGA_VEHICULO 5.0378570 6.6349341 2.20374400 7.1518982 84.5196677
INFRACC_RESUMEN 8.0811364 7.3897426 1.45865212 9.4172296 171.9358202
INFRACC_PEATON -2.8033339 0.9499664 1.67683121 0.1851498 0.9939418
ANIO_MATRICULA_VEHICULO 5.9295547 11.4859615 4.35343730 11.7072978 365.0703755
MES_MATRICULA_VEHICULO 3.4049466 15.7972570 -0.53687395 16.0690088 499.7145951
TIPO_VEHICULO 6.6588133 21.2787095 -0.19610130 20.6711130 199.0782706
ANOMALIA_NINGUNA 9.3328101 6.8778253 5.88453633 8.2228360 143.6617389
ANOMALIA_NEUMATICO 7.8485532 6.8721105 6.70488013 7.4067129 119.1310958
ANOMALIA_REVENTON 6.5487401 5.8373472 4.10660551 6.1963343 74.3142386
ANOMALIA_DIRECCION 5.4273533 4.5115241 3.42907781 4.7797505 66.9991264
ANOMALIA_FRENOS 7.0495934 7.1336800 5.96273787 7.3882427 102.2889357
NUMERO_OCUPANTES_VEH 2.0068816 17.4207823 -2.45100714 17.7556118 140.4688852
MERCANCIAS_PELIGROSAS -5.0237694 -0.6334626 2.38000946 -4.5311280 1.0890107
VEHICULO_INCENDIADO -3.5096342 -0.8431701 4.65061584 -0.4692923 5.3178919
I'd really appreciate any help, comment, or suggestion about the matter at hand or maybe correction about my lack of expression of the problem. Thanks in advance!