2
$\begingroup$

I have been using R's GBM (Gradient Boosting Machine) package for several months. I typically split my data into three partitions: Training, Validation, and Testing. I use the validation data set to appropriately pick the optimal number of iterations. The testing data is a completely untainted data set used for nothing other than final reporting.

I have noticed, however, that the mean error for the validation set at the optimal number of trees is often quite higher than the training data set.

My question is: Do I care that the training and validation error are drastically different? Or do I only care that the validation and testing error are close?

The old guard in my office is convinced that the training and validation error must be similar otherwise the model will not generalize well. For an algorithm like GBM that can perfectly predict training data given enough time, I believe the real assessment of generalization is between the validation and test data sets.

EDIT #1:

I am usually training a model to predict a binary outcome. Therefore the error measurement is binomial deviance. My data sets are large enough that sample size shouldn't be an issue. I typically build on 100k records and ~200 features split into thirds for the train, validation, and test data sets. My target variable is often imbalanced at about 10/90 ratio or even less.

$\endgroup$
  • 3
    $\begingroup$ Could you re-partition the data sets and try again? What are the sizes of your data sets? For relatively small sample sizes this may just be sampling error. You might also want to define want mean error means, I assume you mean Mean Absolute Error or Mean Squared Error? Personally I would take a look at some resampling such as k-fold validation to obtain a slightly more robust estimate. $\endgroup$ – Jonathan Lisic Oct 13 '14 at 12:50
  • $\begingroup$ Am I guessing right that the optimal number of iterations is rather low? $\endgroup$ – cbeleites Oct 14 '14 at 11:45
  • $\begingroup$ @cbeleites On the contrary, I use a very low shrinkage (0.001 or lower) and the optimal number of trees is somewhere between 15,000 and 25,000 $\endgroup$ – Zelazny7 Oct 14 '14 at 12:02
  • $\begingroup$ May be a good idea to increase the shrinkage, as that is a regularisation parameter that ought to have some effect on over-fitting. $\endgroup$ – Dikran Marsupial Oct 14 '14 at 12:12
9
+50
$\begingroup$

If the training error is a lot lower than the test set error, that is usually an indication of over-fitting, but at the end of the day, it is generalisation performance that really matters. Given a choice between a model that has a 0% error on the training set and 20% on the test set, and a model that has a training error of 20% and a test error of 21%, I'll use the former rather than the latter, provided the test set is large enough to be a reliable indicator of generalisation performance.

If you have a problem with more features than training cases, you can always make a linear classifier that is able to classify all of the patterns in the training set without error (provided the points are in general position). In this case it is normal to use regularisation to obtain a classifier with a large margin (c.f. SVM) which improves generalisation. However, you will still end up with a training error of zero, even for a classifier that performs well.

Essentially the training error is potentially misleading and I would recommend most users to ignore it entirely and concentrate on validation set performance (but be aware it is possible to overfit the validation set as well by making lots of adjustments to the model in order to improve validation set performance).

Have you tried using a support vector machine (or some other regularised machine learning method), these are relatively easy to optimise (as there are only a few regularisation and kernel parameters to tune). The GBM looks to me a little tricky to optimise the parameters (e.g. what value of the regularisation parameter to use).

$\endgroup$
  • 4
    $\begingroup$ @downvoter, why the downvote? If the answer is wrong, then please explain why. Kernel machines often give very good results while having a very low error on the training set. $\endgroup$ – Dikran Marsupial Oct 13 '14 at 18:23
  • 1
    $\begingroup$ "usually an indication of over-fitting" ... and overfitting typically means that there is quite a good chance to get better performance with less overfitting methods. Ensemble models can take care of that, but then boosting (in contrast to bagging) is known to be prone to overfitting itself. $\endgroup$ – cbeleites Oct 14 '14 at 11:44
  • 1
    $\begingroup$ Indeed, however the problem is to decide which of the training patterns should be misclassified! There is essentially an estimation problem that means while there may be a better solution to the problem, we may not be able to find it from the limited amount of training data we have. This is why bagging (or other forms of ensemble) is often still helpful even for regularised models. $\endgroup$ – Dikran Marsupial Oct 14 '14 at 12:07
5
$\begingroup$

With such a highly imbalanced $Y$ your effective sample size is likely to be insufficient for splitting the data into 3 samples. You might consider the Efron-Gong 'optimism bootstrap' for validating a full-sample fit. For regression models this is implemented in the R rms package validate functions.

$\endgroup$
  • 1
    $\begingroup$ If there are 100K records and 10/90 ratio then there are still 33K patterns of the minority class in each partition, which seems a pretty reasonable sample size. $\endgroup$ – Dikran Marsupial Oct 13 '14 at 18:25
  • 1
    $\begingroup$ I had read it as 100k for all data combined. If you have training and test samples both larger than 50k with a 0.1 event proportion then you should be ok. $\endgroup$ – Frank Harrell Oct 13 '14 at 19:04
  • $\begingroup$ should be 3.3K minority patterns, sorry been a long day! Still quite a lot by machine learning standards ;o) $\endgroup$ – Dikran Marsupial Oct 13 '14 at 19:09
  • $\begingroup$ That is insufficient for data splitting for achieving reliable results. You will find that results vary too much if you re-split into different halves or thirds of the dataset. $\endgroup$ – Frank Harrell Oct 13 '14 at 19:30
  • 4
    $\begingroup$ I don't understand why someone has downvoted this answer, the suggestion of using boostrapping is a perfectly reasonable one. It is a shame when answers are downvoted without giving a reason so that we can learn from the discussion. $\endgroup$ – Dikran Marsupial Oct 14 '14 at 15:22
1
$\begingroup$

Agree with Jonathan, I would try k-fold validation and compare. With respect to your question though, yes, I think you should care that the training and validation error are drastically different.

Remember, this is not training vs testing; Training and validation essentially come from a qualitatively similar population. Especially with a binary outcome variable, a drastically different error might indicate that the way you split your data has some issues. It might be the case that a lot of Y=1 data points are gathered in the beginning of your dataset and if you just serially split it to 3 parts, you get most of the Y=1 in the training, almost none in the validation, and ... the rest in the test set.

Unless this is a typo and you actually mean training vs testing error, then I would suggest you rethink how you split your data, try k-fold cross validation, bootstrapping or other similar techniques.

$\endgroup$
  • $\begingroup$ I shuffle my data sets before I split them to make sure they are randomly distributed across the partitions. To clarify, I train the model on TRAIN, I tune hyper-parameters, such as the optimal number of trees on VALIDATION, and I test the final model performance on the untainted TEST sample. $\endgroup$ – Zelazny7 Oct 14 '14 at 12:05
  • 1
    $\begingroup$ I think the point may be that stratified sampling might help, where you ensure that each of the three sets have the same proportions of positive and negative patterns in them (which seems a reasonable suggestion for problems with a substantial imbalance). However it could also be that by random chance you have ended up with lots of easily classified patterns in the training set and difficult ones in the validation and test sets. This is unlikely as you have quite a lots of records to use, but cross-validation would guard against this by averaging over many training-validation-test splits. $\endgroup$ – Dikran Marsupial Oct 14 '14 at 12:22
  • $\begingroup$ I tend to use nested cross-validation, where the outer cross-validation is for performance estimation, and in each fold a further cross-validation is performed for tuning the hyper-parameters. $\endgroup$ – Dikran Marsupial Oct 14 '14 at 12:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.