I am performing a hyperparameter grid search for a GBM classifier in H2O, running version 126.96.36.199 (on top of python 3.5.3). This is a multiclass problem (~40 classes).
As a first test, I tried a small grid of parameters, to study the effect of the learning rate,
[0.01, 0.05, 0.1], and max depth,
[5, 7, 10]. In this example, I didn't use a stopping criterion or validation set, but rather set a large number of trees and just let it run.
What strikes me about this behavior is the sudden jump in log-loss at larger tree number for the higher learning rates (and perhaps this behavior would happen eventually for the learning rate of 0.01). My general understanding is that training log-loss should continually decrease with a long tail, while validation log-loss will start to increase at some point, necessitating early stopping to prevent overfitting. Even in that scenario, however, I wouldn't expect such a large and instantaneous jump.
Does anyone have insight into why this behavior might occur? Is there some other parameter that I'm not setting properly that could cause this? Something with the sampling? Or am I fundamentally misunderstanding something about how GBMs function?