2
$\begingroup$

What does it mean if on my dataset, when using leave-one-group-out cross-validation, for some cv folds gradient boosting classifier starts out with a decent validation score (ROC AUC greater than 0.5), but with each iteration the validation performance gets worse rapidly from the very start (ending up with ROC AUC under 0.5)?

Is there anything I can infer about my training set, validation group that was left out, or the set of features from this?

Does this, for example, mean that it is likely that my training set has a significant number of mislabeled examples?

$\endgroup$
4
  • 1
    $\begingroup$ Overfitting is certainly a possibility. Do you mean you have good performance for some iterations then suddenly the performance drop significantly? $\endgroup$
    – SmallChess
    May 3, 2017 at 14:49
  • $\begingroup$ I have good performance on the very first iteration (when a single decision tree is fitted), and on each consequent iteration the performance gets worse steadily and quite rapidly. I basically get the inverse of what I would expect to get. On other validation examples, however, I get the expected behavior (start from a low score, get rapid increases in performance for some iterations, then a gradual dropoff in performance). I am using deep trees at the moment, but I get similar effect even if I use weak learners (in that case, it manifests itself over more iterations). $\endgroup$
    – rinspy
    May 3, 2017 at 15:15
  • $\begingroup$ I suspect I would get an effect like this if my validation set was mislabeled (which is not unlikely), or if the validation group (time series, in my case) has a different relationship between the features and the output variable than the training set. Are these suspicions valid given what I am observing? $\endgroup$
    – rinspy
    May 3, 2017 at 15:20
  • $\begingroup$ Yes - same behavior with max_depth=4, subsample=0.01, colsample_bytree=0.1, learning_rate=0.001. Here is the output: pastebin.com/rgaxn7YA. Here is the output from another CV fold where boosting behaves as expected: pastebin.com/UFexMbqn. What should I make of this and how do I "debug" this further? $\endgroup$
    – rinspy
    May 3, 2017 at 16:07

2 Answers 2

2
$\begingroup$

This is not quite an answer to the question, but yes, turns out I was overfitting after all. I still get the phenomenon I described on some groups, but reducing the variance of the model did help on the majority of groups (in this case, it was by setting the depth of the trees to a low value and setting learning rate, subsample and feature subsampling to lower values than I thought would be reasonable).

$\endgroup$
0
$\begingroup$

Just to add to what you have already mentioned, Leave-One-Out cross validation in itself increases variance. A 10-fold cross validation is generally advised as a decent compromise between bias and variance.

You can also quantify the overfitting measure by evaluation the learning curve where the Y-axis represents both training score and validation score.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.