I've been experimenting with random forests.
I have 30 or so input features and one very right-skewed response.
I've noticed that when the metric for success is the classic
mean squared error, more trees is generally better, which is something you can read all over the place: more trees can't hurt. it can only help.
However, when the metric is
median absolute error (you can see why I might be interested in this metric since my response is so skewed) less trees gives better results!
Reason for problem?
My hunch is that this has something to do with the skewed response variable. The more trees we involve the more likely it is that one of these infrequent but HUGE values gets included in the leaf nodes, thus throwing off the estimate for the majority of inputs that are very low in comparison.
A possible solution is to use the median of leaf nodes as the prediction. The problem is this isn't implemented efficiently anywhere, so experimentation is costly.
Does my reasoning sound valid?