0
$\begingroup$

I have trained h2o.randomforest model on my data and following is the scoring history.

 coring History: 
             timestamp          duration number_of_trees training_MSE     training_logloss training_AUC training_classification_error
1  2015-09-23 03:20:54        19.057 sec               1      0.49373          2.98436      0.89528                       0.18939
2  2015-09-23 03:21:02        26.870 sec               2      0.49292           2.95248      0.90202                       0.18089
3  2015-09-23 03:21:10        34.995 sec               3      0.49328          2.94815      0.90395                       0.17823
4  2015-09-23 03:21:18        43.356 sec               4      0.49291          2.94215      0.90454                       0.17665
5  2015-09-23 03:21:27        51.810 sec               5      0.49304          2.94646      0.90477                       0.17903
6  2015-09-23 03:21:42  1 min  7.623 sec               7      0.49322          2.94274      0.90646                       0.17785
7  2015-09-23 03:22:07  1 min 31.859 sec              10      0.49343          2.94660      0.90674                       0.17827
8  2015-09-23 03:22:39  2 min  4.409 sec              14      0.49347          2.95197      0.90752                       0.17472
9  2015-09-23 03:23:22  2 min 46.928 sec              19      0.49357          2.95385      0.90904                       0.17446
10 2015-09-23 03:24:12  3 min 37.249 sec              25      0.49386          2.96022      0.91061                       0.17231
11 2015-09-23 03:25:12  4 min 37.707 sec              32      0.49395          2.95995      0.91318                       0.16912
12 2015-09-23 03:26:19  5 min 44.072 sec              40      0.49411          2.96468      0.91505                       0.16655
13 2015-09-23 03:27:26  6 min 51.658 sec              48      0.49421          2.96624      0.91729                       0.16374
14 2015-09-23 03:28:41  8 min  6.478 sec              57      0.49431          2.96732      0.91982                       0.16102
15 2015-09-23 03:29:55  9 min 20.593 sec              66      0.49433          2.96980      0.92148                       0.15833
16 2015-09-23 03:31:21 10 min 46.086 sec              76      0.49437          2.97340      0.92272                       0.15528
17 2015-09-23 03:32:44 12 min  9.201 sec              86      0.49445          2.97545      0.92410                       0.15492
18 2015-09-23 03:34:07 13 min 32.039 sec              96      0.49449          2.97710      0.92534                       0.15165
19 2015-09-23 03:34:45 14 min  9.934 sec             100      0.49449          2.97725      0.92576                       0.15191

My question is that intuitively training MSE or training logloss should decrease as trees grow (basically error should decrease) but it is not happening here. Though AUC is increasing and classification error is decreasing as trees grow which is a good thing but why not training MSE or logloss? Is it a concern? Does it mean model is not behaving well on the data set?

Follwing are the parameter settings.

model_rf <- h2o.randomForest(x = predictors, y = labels,
                         training_frame = train_data, 
                         ntree = 100, 
                         mtries = 2,
                         nbins = 50,
                         sample_rate = 0.95,
                         keep_cross_validation_predictions = T,
                         binomial_double_trees = F,
                         min_rows = 5000,
                         balance_classes = T,
                         max_depth = 12)
                         #nfolds = 2)

My problem is a classification problem.

If any random forest guru could help me out here, that would be great!

$\endgroup$
1
$\begingroup$

You have a lot of atypical parameters here and I believe it is causing a poor-fitting model that exhibits random performance oscillation.

Though first notice that the AUC is increasing. So one answer to the question is that it is possible and even reasonable to have classification tradeoffs that yield decreasing performance by one metric and increasing by another.

However, the oscillating performance may be the result of the parameter choices. If so, the parameter most likely to cause the performance issues is the mtries value of 2. This means that for each of the 100 trees you are building, a random 2 columns are picked for the entire tree. So it's quite likely that you get a poor model with just two columns, for most of those trees. If it happened to randomly choose two of the more powerful columns in a few of your first few models, it is very reasonable to get the output progression you have shown. The default is 1/3 of the columns for classification problems. The square root of the number of columns is a common default as well (used for regression problems in H2O).

min_rows of 5000 is also very high. The default is 10, using up to 30 is fairly common. This means that the algorithm will avoid potentially useful splits of the data set if either of the resulting nodes has fewer than 5000 observations. At a depth of 12, you'd have to have over 20 million rows even with perfect splits.

A nice feature of random forest is that the default parameters are fairly standard across implementations (e.g. R, scikit, H2O). I would suggest trying those out first.

In general random forests improve as you add trees, as you expect. But each tree is independent, so I think you randomly got some more powerful columns sampled early.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.