22
$\begingroup$

I have historic sales data from a bakery (daily, over 3 years). Now I want to build a model to predict future sales (using features like weekday, weather variables, etc.).

How should I split the dataset for fitting and evaluating the models?

  1. Does it need to be a chronological train/validation/test split?
  2. Would I then do hyperparameter tuning with the train and validation set?
  3. Is (nested) cross validation a bad strategy for a time-series problem?


EDIT

Here are some links I came across after following the URL suggested by @ene100:

  • Rob Hyndman describing "rolling forecasting origin" in theory and in practice (with R code)
  • other terms for rolling forecasting origin are "walk forward optimization" (here or here), "rolling horizon" or "moving origin"
  • it seems that these techniques won’t be integrated into scikit-learn in the near future, because “the demand for and seminality of these techniques is unclear” (stated here).

And this is another suggestion for time-series cross validation.

$\endgroup$
8
$\begingroup$

This link from Rob Hyndman's blog has some info that may be useful: http://robjhyndman.com/hyndsight/crossvalidation/

In my experience, splitting data into chronological sets (year 1, year 2, etc) and checking for parameter stability over time is very useful in building something that's robust. Furthermore, if your data is seasonal, or has another obvious way to split in to groups (e.g. geographic regions) then checking for parameter stability in those sub-groups can also help determine how robust the model will be and if it makes sense to fit separate models for separate categories of data.

I think that statistical tests can be useful but the end result should also pass the "smell test".

$\endgroup$
3
$\begingroup$

1) Technically speaking, you don't need to test out of sample if you use AIC and similar criteria because they help avoid overfitting.

3) I don't see how you can do the standard CV because it implies training a time series model with some missing values. Instead, try using a rolling window for training and predict the response at one or more points that follow the window.

$\endgroup$
  • 3
    $\begingroup$ AIC in time series is often based on one period ahead forecast error. It does not tell much about dynamic n-period ahead forecast performance. Hence, it's absolutely not enough. $\endgroup$ – Aksakal Jan 2 '15 at 0:29
  • $\begingroup$ Could you provide a reference? $\endgroup$ – James Jan 13 '15 at 22:52
1
$\begingroup$

I often approach problems from a Bayesian perspective. In this case, I'd consider using overimputation as a strategy. This means setting up a likelihood for your data, but omit some of your outcomes. Treat those values as missing, and model those missing outcomes using their corresponding covariates. Then rotate through which data are omitted. You can do this inside of, e.g., a 10-fold CV procedure.

When implemented inside of a sampling program, this means that at each step you draw a candidate value of your omitted data value (alongside your parameters) and assess its likelihood against your proposed model. After achieving stationarity, you have counter-factual sampled values given your model which you can use to assess prediction error: these samples answer the question "what would my model have looked like in the absence of these values?" Note that these predictions will also inherit uncertainty from the uncertainty present in coefficient estimates, so when you collect all of your predicted values for, e.g. March 1, 2010 together, you'll have a distribution of predictions for that date.

The fact that these values are sampled means that you can still use error terms that depend on having a complete data series available (e.g. moving average), since you have a sampled outcome value available at every step.

$\endgroup$
1
$\begingroup$

In your case you don't have a lot of options. You only have one bakery, it seems. So, to run an out-of-sample test your only option is the time separation, i.e. the training sample would from the beginning to some recent point in time, and the holdout would from that point to today.

If your model is not time series, then it's a different story. For instance, if your sales $y_t=f(t)+\varepsilon_t$, where $f(t)$ is a function of different exogenous things like seasonal dummies, weather etc. but not $y_{s< t}$, then this is not a dynamic time series model. In this case you can create the holdout sample in any different ways such as random subset of days, a month from any period in the past etc.

$\endgroup$
  • $\begingroup$ Sounds promising. What is the subscript "s" in y_s<=t? $\endgroup$ – tobip Jan 2 '15 at 1:08
  • $\begingroup$ If you're forecasting for time $t$, then you're not using any of $y_s$, where $s<t$. $\endgroup$ – Aksakal Jan 2 '15 at 1:09
  • $\begingroup$ I guess my bakery sales rather depend on exogenous factors. I don't think it's a strictly dynamic time series model as, e.g., stock prices. In my data, the Mean Average Error with a usual 5-fold CV is only slightly larger than a 5-fold time-series CV that preserves the chronological order of the folds as described here (e.g., 30.7 vs. 33.8 for an ExtraTreesRegressor). Not sure if this is a valid way to empirically test for a dynamic model? $\endgroup$ – tobip Jan 2 '15 at 1:42
  • $\begingroup$ Something like ARIMA would be a dynamic model. The sales are usually persistent, i.e. today's sales are like yesterday's. $\endgroup$ – Aksakal Jan 2 '15 at 2:26
0
$\begingroup$

Disclaimer: The method described here is not based on thorough reading of the litterature. It is my best attempt at improvising a K-fold CV method for a multivariate timeseries analysis with relatively short input window lengths (assuming no/low dependence over longer time spans), where there was a problem with non-homogeneous presence of data sources over the data collection period.

First the series of observations is transformed into a series of observation history windows of length window_length and with step 1 between windows (no stride). Then the principle is to split the window dataset in "shards" many times longer than window_length (but many more than the number of model instances), and deal out the shards (like playing cards) as validation data to separate model instances. To keep the models more cleanly separated, a quarantine window of window_length at the beginning of each shard is held out of all training.

The models are trained on all shards except their own, and validation is done on their own shards. Validation of the collection/ensemble of models is done by summing the validation error over all shards, where each shard is handled by its corresponding submodel. Testing on unseen data can be done using an average (or other suitable combination) of the outputs of all the trained models.

This method is intended to reduce dependence on the system (and data sources) being the same over the entire data collection period. It is also intended to give every rough part of the data the same influence on the model. Note that to not allow the quarantine windows to harm training, it is a point that the shard length does not align too well with periods that (are expected to) appear in the data, such as (typically) daily, weekly and yearly cycles.

The ensemble model may perhaps handle completely new data badly. (I don't know yet.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.