I am applying machine learning (XGBoost) to certain problem regarding time series classification, as input as uses some numerical values around 200 features and vectorized text (tfidf).

The result I get are a bit confusing - ROC AUC highly dependent on the way the data is split.

In case of random split (tried different seeds) results are around AUC=0.70 (for validation set). However if I split data so it represents characteristic periods of that time series (say time-series going up split into 2 sets train and val, going down and sideways the same, and then merge those respective sets) I get AUC=0.52 at best.

What may cause that behaviour ? Any ideas ?

  • 2
    $\begingroup$ This raises many issues. I'll just mention one: data splitting is an arbitrary, low-precision approach to model validation. Consider the Efron-Gong optimism bootstrap as implemented in the R rms package, or use 100 repeats of 10-fold cross-validation, averaging over the 100. $\endgroup$ – Frank Harrell Apr 9 '16 at 16:15

I don't think there's anything especially remarkable going on. Statistics obtained from train-test splits are subject to random variation, just like every other statistic. If the variation is large, it's because the underlying process has high variance.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.