4
$\begingroup$

To improve my intuition on shrinkage models, I want to "recode" the lasso by myself. However, I'm at the point, where I have to program the k-fold Cross-Validation. At my future application of the algorithm I will use time series models, especially (V)AR(p). Because of the time-ordering, the random k-fold- Validation" should cause into problems, right ? Regarding to that I've two questoins:

1) How the glmnet package is dealing with the cv of AR models

2) Do you know a way some sources, that show me how I can deal with it ?

I found this in the blog of hyndman

$\endgroup$
1
  • $\begingroup$ As far as I know, the glmnet package doesn't deal with AR models at all (neither "AR" nor "time series" are mentioned in the documentation). $\endgroup$ Jan 2, 2021 at 17:40

1 Answer 1

1
$\begingroup$

You need to use expanding window or rolling window, consider below example:

Lets say, you have a data-set of 100 obervations and you need at least 10 observations to fit a model, and you are estimating an AR(1) model. Then your algorithm will look like this:

FEATURES = X = [y_1, y_2, y_3, ...., y_99]

TARGET = Y = [y_2, y_3, y_4, ...., y_100]

CV_SCORES = []

FOR n in [10, 99) DO:

    train_X = [y_1, y_2, ..., y_n]

    train_Y = [y_2, y_3, ..., y_{n+1}]

    test_X = [y_{n+1}]

    test_Y = [y_{n+2}]

    fit your model to (train_X, train_Y)

    predict with test_X

    calculate the score with prediction and test_Y

    append the score to CV_SCORES
END

The methodology above is expanding window and it is time-series version of leave-one-out cross validation.

$\endgroup$
2
  • 1
    $\begingroup$ Although it is definitely valid and standard to do so, you actually do not need to use expanding or rolling windows; regular K-fold cross validation is valid for AR models under certain conditions. Bergmeir, Hyndman and Koo (2015) shows this (it is linked in the blog post in the question). $\endgroup$
    – Chris Haug
    Dec 31, 2020 at 15:08
  • $\begingroup$ I was not aware of this paper thank you so much. I gave AR as an example, as far as I understand from the paper, if the correct process is not AR and the error terms are correlated, you still carry the risk of overfitting. Thus I would still use expanding or rolling window since that methodology is more close to how the model works in production, and it is not dependent on the model specification. $\endgroup$ Dec 31, 2020 at 17:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.