# Fitting an ARIMAX model with regularization or penalization (e.g. with the lasso, elastic net, or ridge regression)

I use the auto.arima() function in the forecast package to fit ARMAX models with a variety of covariates. However, I often have a large number of variables to select from and usually end up with a final model that works with a subset of them. I don't like ad-hoc techniques for variable selection because I am human and subject to bias, but cross-validating time series is hard, so I haven't found a good way to automatically try different subsets of my available variables, and am stuck tuning my models using my own best judgement.

When I fit glm models, I can use the elastic net or the lasso for regularization and variable selection, via the glmnet package. Is there a existing toolkit in R for using the elastic net on ARMAX models, or am I going to have to roll my own? Is this even a good idea?

edit: Would it make sense to manually calculate the AR and MA terms (say up to AR5 and MA5) and the use glmnet to fit the model?

edit 2: It seems that the FitAR package gets me part, but not all, of the way there.

• I'd be curious to know if you ever managed to do something about this problem. I had it marked for any update - so just touching base here. Aug 16, 2012 at 20:54
• Never manged to solve this. I thought about writing my own implementation but didn't get around to it. Maybe you could manually calculate the AR and MA terms and then apply glmnet to that data?
– Zach
Aug 16, 2012 at 23:02
• Hi Zach, I wondered if you have taken this any further or whether it is still parked. I have a need for such an algorithm to forecast a large number of series. Nov 28, 2014 at 17:11
• @MattWeller I talked a bit with Rob Hyndman, the author of the excellent forecast package for R. He said it would be difficult with the full ARIMA, because you'd have to wrap the lasso around the nonlinear ARIMA optimizer. One partial solution would be to fit an AR model using glmnet with lagged variables. As far as I know, no one's done this with a full ARIMA model yet.
– Zach
Nov 28, 2014 at 21:28

This is not a solution but some reflections on the possibilities and difficulties that I know of.

Whenever it is possible to specify a time-series model as $$Y_{t+1} = \mathbf{x}_t \beta + \epsilon_{t+1}$$ with $\mathbf{x}_t$ computable from covariates and time-lagged observations, it is also possible to compute the least-squares elastic net penalized estimator of $\beta$ using glmnet in R. It requires that you write code to compute $\mathbf{x}_t$ to form the model matrix that is to be specified in glmnet. This works for AR-models but not directly for ARMA-models, say. Moreover, the cross-validation procedures of glmnet are not sensible per se for time-series data.

For more general models $$Y_{t+1} = f(\mathbf{x}_t, \beta) + \epsilon_{t+1}$$ an implementation of an algorithm for computing the non-linear least-squares elastic net penalized estimator of $\beta$ is needed. To the best of my knowledge there is no such implementation in R. I am currently writing an implementation to solve the case where
$$Y_{t+1} = \mathbf{x}_t g(\beta) + \epsilon_{t+1}$$ the point being that it is paramount for model selection that the lasso penalization is on $\beta$ and not $g(\beta)$. If I recall the ARIMA-parametrization correctly it also takes this form $-$ but I cannot offer any code at the moment. It is (will be) based on A coordinate gradient descent method for nonsmooth separable minimization.

Another issue is the selection of the amount of penalization (the tuning parameters). It will generally require a form of cross-validation for time-series, but I hope to be able to work out some less computationally demanding methods for specific models.

• The caret package for R has a good implementation of time series cross validation that could be used to select the parameters for glmnet. Thanks for the answer.
– Zach
Feb 18, 2013 at 15:53

I was challenged by a client to solve this problem in an automatic i.e. turnkey way. I implemented an approach that for each pair ( i.e. y and a candidate x ) , prewhiten , compute cross-correlations of the pre-whitened series, identify the PDL ( OR ADL AUTOREGRESSIVE DISTRIBUTED LAG MODEL including any DEAD TIME ) while incorporating Intervention Detection to yield robust estimates, develop a "measure" for this structure. After conducting this for ALL candidate regressors, rank them by the "measure" and then select the top K regressors based upon the "measure". This is sometimes referred to as Linear Filtering. We successfully incorporated this heuristic into our commercially available time series package. You should be able to "ROLL YOUR OWN" if you have sufficient time and statistical programming skills and/or available modules to implement some of these tasks.

• I specifically want to incorporate the lasso or elastic net into an ARIMA model.
– Zach
Oct 7, 2011 at 15:37
• I'm not really looking for a linear filter (though this is useful). I specifically want a way to incorporate the lasso, ridge regression, or elastic net into an ARIMAX model.
– Zach
Jan 12, 2012 at 16:11
• At the moment I'm not interested in level shifts, local time trends, coefficient shifts, or non-constant error variance. I'm very specifically interested in applying lasso, ridge, or elastic net regularization to ARIMAX models.
– Zach
Feb 15, 2013 at 14:37
• If you can incorporate lasso, ridge, or elastic net regularization into the above methodology and post some example code in an open source programming language, I'd be happy to award you the bounty.
– Zach
Feb 16, 2013 at 17:09
• @frank you are trying to do identification via estimation (step-down) and that doesn't work because there are toooooo many combinations of possible pulses, level shifts , seasonal pulses and time trends.,. If these structures are needed but ignored , parameter estimation is biased and parametric tests of significance go out the window. Jul 5, 2019 at 19:32