This question's context is time series forecasting using regression, with multivariate training data. With a regularization method like LARS w/ LASSO, elastic net, or ridge, we need to decide on the model complexity or regularization parameters. For example, the ridge $\lambda$ penalty or the number of steps to go along the LARS w/ LASSO algorithm before hitting the OLS solution.

My first instinct is to use cross-validation to infer a decent value of the regularization parameter. For LARS w/ LASSO, I would infer the (effective) degrees of freedom that optimizes some fitness function like $\frac{1}{n}\sum_{i{\le}n}|\hat{y}_i-y_i|$. However with time series data, we should cross-validate out-of-sample. (No peeking into the future!) Say there are two feature time series $x_1$ and $x_2$ and I am forecasting a time series $y$. For each step of time $t$, train with $x_{1,1}$ through $x_{1,t}$ and $x_{2,1}$ through $x_{2,t}$ — and then forecast $\hat{y}_{t+1}$ and compare with the actual $y_{t+1}$.

This framework makes sense from an out-of-sample perspective, but I worry that earlier cross-validation steps (low $t$) will be overemphasized when averaging over all the equally-weighted steps. Should the first few time series cross-validation steps, the ones that use much less training data, be suppressed when inferring (regularization) model parameters? I might prefer a model complexity (regularization) level that "did better" on those later cross-validation steps using more training data.

| cite | improve this question | | | | |

You can include a "minimum" number of observations that you think you need to fit your model, and exclude n< this number from cross validation. Obviously, you can't fit a model using just the 1st sample, and you can't really fit a model using the 1st 2 samples. At some reasonable point (5? 10?) you'll have enough observations to fit a valid model, so start at that point.

| cite | improve this answer | | | | |
  • $\begingroup$ Thanks, Zach. Your minimum example count has been my best solution so far, but it still feels a bit hack-y. $\endgroup$ – someben Jun 9 '11 at 14:25
  • $\begingroup$ @someben: Rob Hyndman recommends this approach on his site. If you scroll to the bottom, you can see him talking about t=m,..n-1, where m is the minimum number of observations you need to fit the model. robjhyndman.com/researchtips/crossvalidation $\endgroup$ – Zach Jun 9 '11 at 14:40
  • $\begingroup$ @someben And yes, it feels "hacky" but i haven't found many other methods to cross-validate time series. Another idea is to use a "blocked bootstrap" method to re-sample your time series and create a new series with similar statistical properties. This only works if your series is stationary, and is probably trickier to implement correctly than a simple time-series cross-validation. en.wikipedia.org/wiki/… $\endgroup$ – Zach Jun 9 '11 at 14:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.