I have been using R's GBM (Gradient Boosting Machine) package for several months. I typically split my data into three partitions: Training, Validation, and Testing. I use the validation data set to appropriately pick the optimal number of iterations. The testing data is a completely untainted data set used for nothing other than final reporting.
I have noticed, however, that the mean error for the validation set at the optimal number of trees is often quite higher than the training data set.
My question is: Do I care that the training and validation error are drastically different? Or do I only care that the validation and testing error are close?
The old guard in my office is convinced that the training and validation error must be similar otherwise the model will not generalize well. For an algorithm like GBM that can perfectly predict training data given enough time, I believe the real assessment of generalization is between the validation and test data sets.
I am usually training a model to predict a binary outcome. Therefore the error measurement is binomial deviance. My data sets are large enough that sample size shouldn't be an issue. I typically build on 100k records and ~200 features split into thirds for the train, validation, and test data sets. My target variable is often imbalanced at about 10/90 ratio or even less.