As part of some preliminary research, I'm experimenting with a random forest classification model for predicting whether the S&P 500 will be higher or lower at tomorrow's close versus today's close.

My data are structured as follows:

  • y.shape => (5000, 1) / an array of 1s and 0s representing whether tomorrow's close price is higher (1) or lower (0)
  • X.shape => (5000, 20) / 20 features, mostly interval and ordinal data

If I use sklearn's handy traintestsplit with a testsize of 20% and shuffle=True (the default), then I get a reasonably good k-fold CV score (GridSearchCV().bestscore_) and a test score above 70%.

If I change to shuffle=False, both the CV score and test score drop to around 55%.

I'd really like to get a better understanding of why this is happening. Of course it could just be a coincidence, but I have read a lot of talk about shuffle=True being a bad idea for evaluating time series prediction models.

Could anyone shed some light on this?

In particular, what I really struggle with is… why does it really necessarily matter? Is it not possible that in my model, every observation or row can be looked at independently of all others? For example, let's say that in my features I have one that refers to the price of the day before, and another one that refers to the day of the week - with that kind of set of features, surely it's reasonable to look at every observation independently since the temporal elements are (at least partially) captured in the features?

And ultimately, if you're getting let's say 75% in the shuffle=True version, surely that speaks for itself? It's not like this result is biased in some way, like if one of your features was the price of tomorrow.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.