4
$\begingroup$

Hyndman's great explanation of proper time series CV is at the bottom of the page in the following link: http://robjhyndman.com/hyndsight/crossvalidation/

Leave-One-Out illustration in the following link: http://i.imgur.com/qrQI4LY.png

In the 'leave one out' illustration if the dataset in the illustration was time series data and was sorted from past to present from left to right, wouldn't it be identical to Hyndman's explanation on Time Series CV? If not, how so?

$\endgroup$
5
$\begingroup$

The short answer is if you used leave-one-out CV for time series, you would be fitting model parameters based on data from the future. The easiest way to see how this is to just write out what both procedures look like using that same data. This makes the difference glaringly obvious. Following Hyndman's notation let $y(1), \ldots, y(T)$ be the time series and $m$ the minimum number of points need to build a model. Then the procedure described by Hyndman works as follows

For t = m to T-1:
    Fit model with y(1), ... y(t)
    e(t+1) = y(t+1) - y*(t+1)
Calculate MSE of e(m+1) to e(T)

For leave-one-out CV the procedure look like

For t = 1 to T:
    Fit model with y(1), ..., y(t-1), y(t+1), ..., y(T)
    e(t) = y(t) - y*(t)
Calculate MSE of e(1) to e(T)

Notice how in the time series version we're actually using a different number of points to fit each model, namel $m, m+1, \ldots, T-1$. Compare this to the other version where one is always using $T-1$ points.

$\endgroup$
  • $\begingroup$ Thanks! Would be nice to have some time series cv code or packages in python. Sklearn doesn't offer it. $\endgroup$ – mlo Nov 14 '13 at 14:40
4
$\begingroup$

The explanations are both right, but they are for different situations. As usual, it all boils down to the question how to obtain statistically independent splits of your data.

  • The image you linked and your description is for a situation where you have repeated measurements of time series.
    In this situation you can leave out complete time series from your data. Imagine you want to predict some property based on a new complete measurement of another time series, e.g. classification of EEG readings. You can assume EEG readings of different patients to be statistically independent and a scenario where only complete readings are used is sensible. In that case the natural way of splitting the data would be by patient.

  • Hyndman discusses a situation where you essentially have only one (ongoing) measurement of a time series, and you want to predict future values of the time series from past measurements. Thus, you split by time, and the future implies that none of the following time points is known.
    In the EEG example, this corresponds to trying to predict what the next seconds/minutes of the EEG of the given patient would be.
    This type of splitting is also important when you want to measure how long a model is valid, see e.g. Esbensen, K. H. and Geladi, P.: Principles of Proper Validation: use and abuse of re-sampling for validation, J Chemom, 2010, 24, 168-187.

  • Another situation where you'd need to split by time and also by case would be: imagine you'd like to do predictions on future value of stocks. Again, you need to split by case (stock). But of course, the tested stock's value at a given time may (is probably) also be correlated with the value of other stocks at that time. Thus, you also need to leave out all "future" data of all stocks from model training.

$\endgroup$
  • $\begingroup$ Sounds like it would take a very long time to have the model fit then predict then fit then predict over the course of say 30 years of market data for one individual stock. But I guess that's what needs to be done to implement the proper time series cv needed to get realistic feedback. $\endgroup$ – mlo Nov 14 '13 at 14:39
  • $\begingroup$ @mio: you don't necessarìly need to do it with every time point. Instead, you could e.g. randomly select 1000 time points an train models each up to the particular time point and test with the time points after the chosen one. That would be a time-series versiion of repeated/iterated set validation. $\endgroup$ – cbeleites Nov 15 '13 at 21:43
  • $\begingroup$ using your example, say you use those randomized 1000 time points and say you pick a minimum observation to test on of 800 time points and use the remaining 200 time points to test then train then test then train. Wouldn't you think that doing that process 200 times be computationally time consuming when say your using a Random Forest Classifier with 10,000 estimators for example? $\endgroup$ – mlo Nov 15 '13 at 22:04
  • $\begingroup$ @mlo: actually I was suggesting to do 1000 tests and train 1000 models on times before the selected cut time and test on the time points afterwards. This is of course 1000 x the computational effort compared to calculating one model (and not properly testing it). But there is no difference in the effort to 100 iterations of a 10-fold cross validation of the row-wise splitting scenarios. (and you could run e.g. only 25 such sets for a start and then decide how many you are going to need) $\endgroup$ – cbeleites Nov 16 '13 at 8:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.