1
$\begingroup$

I have been tasked to put together a monthly forecast model. I'm hoping to use some of the models in scikit-learn for this.

My original dataset is daily level and very rich. If I aggregate it to month-level, this gives me only 24 usable observations so many models may struggle with that. It feels like I should be able to make more use of my richer, daily dataset for my problem.

I have heard somewhere (but can't remember where or whether I imagined it!) that a workaround is to create "fake" monthly data by creating rolling sums say from 26th Dec to 26th January. So for December I would have 31 "fake months", one starting on each day of December and ending on the corresponding day number in January.

So for December I'd have 31 entries:

1st Dec: aggregated data for 1st Dec - 1st Jan

2nd Dec: aggregated data for 2nd Dec - 2nd Jan

...

31st Dec: aggregated data for 31st Dec - 31st Jan

Specifically, is this a valid workaround to artificially increase sample size in short time series in order to train machine learning models? I can see straight off the bat that autocorrelation is a massive issue and likely falls short when next month has fewer number of days.

Are there any other workarounds for working with short time series beyond "getting more data" which I can't. Have googled but nothing came up -- I think I am lacking the necessary vocabulary?

Thank you

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ What about modelling the daily data directly without aggregating to a lower frequency? Have you considered that? You could forecast the daily series and sum the forecasts to obtain monthly sums where needed. However, whichever way you do it, there is no workaround to the fact that you only have two observations of seasonality for each month/day (two full cycles). $\endgroup$
    – Richard Hardy
    Commented Jan 26, 2017 at 18:55

1 Answer 1

Reset to default
2
$\begingroup$

  1. Using real data to generate simulated data doesn't add new information beyond the original, real data (and simulation parameters).

    • Imagine my sister has read all the existing Harry Potter books and would like to read another. Will she learn anything if I photocopy an existing Harry Potter, rearrange the chapters, and staple it all back together? I haven't added any new information.

  2. If you're concerned that aggregating daily data at the monthly level destroys too much information, then work at the daily level. (I agree with @Richard Hardy in the comments.)

    • Be aware that depending on the problem, there might not be much additional useful information in daily data compared to monthly data. For example, when estimating returns (eg. stock returns, bond returns), the statistical power comes from the length of time observed, not the frequency of the observation.

There's no magic shortcut to get more data. With small samples, your best hope is to try to use what data you have more efficiently and possibly rely more on priors, theory, etc...

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.