# AUC (and other measures) dependent on the way data is split

I am applying machine learning (XGBoost) to certain problem regarding time series classification, as input as uses some numerical values around 200 features and vectorized text (tfidf).

The result I get are a bit confusing - ROC AUC highly dependent on the way the data is split.

In case of random split (tried different seeds) results are around AUC=0.70 (for validation set). However if I split data so it represents characteristic periods of that time series (say time-series going up split into 2 sets train and val, going down and sideways the same, and then merge those respective sets) I get AUC=0.52 at best.

What may cause that behaviour ? Any ideas ?

• This raises many issues. I'll just mention one: data splitting is an arbitrary, low-precision approach to model validation. Consider the Efron-Gong optimism bootstrap as implemented in the R rms package, or use 100 repeats of 10-fold cross-validation, averaging over the 100. – Frank Harrell Apr 9 '16 at 16:15

## 1 Answer

I don't think there's anything especially remarkable going on. Statistics obtained from train-test splits are subject to random variation, just like every other statistic. If the variation is large, it's because the underlying process has high variance.