Here's my situation: Binary classification and I've got a training set of roughly 250k samples and 10 features, and a validation set of roughly 100k with the same number of features. I'm fitting GBDT to the data with subsampling, so BGBDT I guess?
Anyhow I took the training set and split it 80/20 and did a grid-search over the parameter space (using cross-validation) on the 80% and then fit a new model to the 20% using the paramaters obtained from the grid-search and used log loss to determine the error. The 20% are held-out and are trained separately from the 80%. Ok fine. I'm getting good results wrt log loss on the held-out model for the 20%, and I'm monitoring the out of bag estimate of the deviance during the tree building process which is decreasing throughout the run.
Overall I'm training roughly 150 trees, subsampling at 0.25, with a learning rate of 0.1, and maximum depth of 10. The results from training the held-out samples this way give good results... if there was a problem with overfitting I would expect that the held-out samples wouldn't perform well. But when I retrain the model on the entire training set and then predict on the validation set my results are very poor. One thing to note, the training and validation sets are uniformly sampled from the same distribution and the classes are balanced.