I've always subscribed to the folk wisdom that decreasing the learning rate in a gbm (gradient boosted tree model) does not hurt the out of sample performance of the model. Today, I'm not so sure.
I'm fitting models (minimizing sum of squared errors) to the boston housing dataset. Here is a plot of error by number of trees on a 20 percent hold out testing data set
It's hard to see what's going on at the end, so here's a zoomed in version at the extremes
It seems that in this example, the learning rate of $0.01$ is best, with the smaller learning rates performing worse on hold out data.
How is this best explained?
Is this an artifact of the small size of the boston data set? I'm much more familiar with situations where I have hundreds of thousands or millions of data points.
Should I start tuning the learning rate with a grid search (or some other meta-algorithm)?