2
$\begingroup$

I apply the xgboost algorithm for classification. I perform cross-validation in the training data set in order to find parameters (eta, step size shrinkage, = 0.01, maximum depth of a tree: 14, 1400 rounds) for best accuarcy and I get something like 0.9. However on the test data set I get 0.5.

Furthermore my prediction in-sample looks like this:

enter image description here

Using classical methods (glm for example) the probabilities are much more "unclear" meaning that they cluster around 0.5. In the case of xgboost I get a much more spread-out picture. Is this a sign of overfitting? Which parameters can I calibrate to avoid this? I assume gamma is the one, I use the default 0. What are typical values for gamma?

$\endgroup$
2
$\begingroup$

GBT's and RF's tend to not need that much parameter tuning. To see this much of a performance difference I'd suspect something else.

I'd ask:

  1. Did you do any feature selection or engineering on the training data but outside the CV loop? That is the biggest culprit in models that don't generalize?
  2. Is the data highly dimensional and noisy enough that there are are likely to be features that could legitimately perform well on the training data but not the test? This is really common in genetic data with 10's of thousands of features and only hundreds of cases.
  3. Is there some sort of unique identifier or leaked data in the training set that allows a good fit?
  4. Is there some sort of batch effect or shifting covariates? Ie this is really common if the hold out set is newer data then the training as is common in ongoing studies.
$\endgroup$
  • $\begingroup$ Thanks, I will check 1) of your list first. In order to get the input for xgboost I have to generate the model matrix manually and I can imagine that with approx 70 features and some factors the model matrix could be somewhat incomplete. $\endgroup$ – Ric Dec 13 '15 at 10:10
  • $\begingroup$ Yeah manual data prep is really error prone, something even something like centering, scaling or encoding features outside of the CV loop and leak information about the data set as a whole. Its best to do the absolute minimum needed to get to the point that you can generate your folds and then do all of your preprocessing on each fold and then again on all of the data used to train the model for validation. You want to validate the whole training data and model generating process not just xgboost. $\endgroup$ – Ryan Bressler Dec 14 '15 at 18:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.