I'm having trouble getting good scores on cross validated metrics on time series regression models. Essentially, I am trying to model product purchases based on amount of money spent on different advertising methods. I only have 42 weeks of valid data for the model, which I know is not ideal but I cannot get more.

In addition to the data being relatively small, it is also highly seasonal. The purchases follow a curve that fits well overall to a cos function with a period of 52 weeks (one year).

For cross validation, I am using an expanding window approach similar to the one outlined here: https://medium.com/eatpredlove/time-series-cross-validation-a-walk-forward-approach-in-python-8534dd1db51a

So, I can get good fits to the data overall, especially when using a pair of first order fourier terms to control for seasonality. However, the cross val performance is terrible, I'm getting models with negative R2 values. My theory is that because that data is relatively small, if one of the fold is over a hump of the curve in purchases or is a part where the curve is trending up, the learned parameters from the previous portion will not have accounted for this and make bad predictions.

So my question is, is my theory valid, and if so, is there any thing I can do to get better performance on a cross validation methodology? If I cannot, is it "valid" to not use cross validation and use another method to validate fit, maybe a test train split?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.