I am applying machine learning (XGBoost) to certain problem regarding time series classification, as input as uses some numerical values around 200 features and vectorized text (tfidf).
The result I get are a bit confusing - ROC AUC highly dependent on the way the data is split.
In case of random split (tried different seeds) results are around AUC=0.70 (for validation set). However if I split data so it represents characteristic periods of that time series (say time-series going up split into 2 sets train and val, going down and sideways the same, and then merge those respective sets) I get AUC=0.52 at best.
What may cause that behaviour ? Any ideas ?