# Variable selection in time-series forecasting

I have a time-series forecasting task and would like some input on variable selection and regularisation.

My problem has the following characteristics:

• 2,000,000 sample size.
• Most of the time, no change in the response. Change in the response happens 0.3% of the time and is what I'm interested in (don't talk about asymmetric loss functions and the like - I'm purely interested in variable selection/regularlisation).
• I will be predicting direction (up/down/no change).
• 100 predictor variables. Mostly hand engineered from complex non-stationary data. All carry some predictive power - the useless ones with no univariate forecasting power were eliminated early in the process.
• There will be clusters of highly correlated predictors - in general if you select two predictors at random the correlation will not be high.
• Predictors are heteroscedastic in nature.
• Some predictors are heavily serially correlated, many are not.
• Predictors all evolve continuously and are decimal processes. They are all bell shaped.
• Some could be non-stationary, but are thought to carry critical information about the evolution of the response process that would be lost upon differencing or similar $I(0)$ transforms.
• My forecasting model will be linear, but least squares loss function is not certain (perpendicular regression and quantile regression and discrete response regression will be tried).
• Standard error unbiasedness and variance is not a direct consideration. All I care about is out of sample forecast accuracy and model generalisation.

I know I can use anything from non-negative garotte, ridge, lasso, elastic nets, random subspace learning, PCA/manifold learning, least angle regression and various yucky hacks to select based on out of sample forecast performance. But is there anything specific to forecasting, or to the characteristics of my data, that would push me one way over another (time efficiency is important - trying everything and selecting the best is not practical).

Note that decision boundaries on the forecasts are not intended to be part of the regularisation algorithm, but integration could be possible.

I will also be using a supercomputer.

• Just want to clarify the second point. By saying that change in response happens 2% of the time, you mean that your time series which you want to forecast contains only 2% non-zero values? Oct 3 '13 at 14:18
• @mpiktas Please see my update. But your interpretation is correct, except it's 0.3% Oct 3 '13 at 14:23

• I don't understand the second point. I have some function of the features $f(X(t)) = Y(t+1)$ that I need to estimate. Once I have $\hat{f}$ then I only need information at time $t$ to get my forecast -- no future information needed. Oct 4 '13 at 9:56
• Yes, but in the usual regression it is assumed that $Y_t=f(X_t)$. Writing $Y_{t+1}=f(X_t)$ assumes that there is no contemporaneous effect, which might be false. Although for forecasting problem with huge sample size this is assumption can be made. Oct 4 '13 at 10:42
• If you use the model $Y_{t+1}=f(X_t)$ and are interested in one step ahead forecasts then it is ok. More common situation is the model $Y_t=f(X_t)$. When you divide into in-sample and out-of-sample and measure the forecasting performance you can use $X_t$ from the out-of-sample to get $\hat{Y_t}$ and then look at MAE, MAPE, etc. But that would not be right, since in real life exercise $X_t$ are unavailable. Hence although you have out-of-sample $X_t$ you need to forecast them and use only out-of-sample $Y_t$ to measure forecasting performance. Oct 4 '13 at 11:21