I have a time-series forecasting task and would like some input on variable selection and regularisation.

My problem has the following characteristics:

  • 2,000,000 sample size.
  • Most of the time, no change in the response. Change in the response happens 0.3% of the time and is what I'm interested in (don't talk about asymmetric loss functions and the like - I'm purely interested in variable selection/regularlisation).
  • I will be predicting direction (up/down/no change).
  • 100 predictor variables. Mostly hand engineered from complex non-stationary data. All carry some predictive power - the useless ones with no univariate forecasting power were eliminated early in the process.
  • There will be clusters of highly correlated predictors - in general if you select two predictors at random the correlation will not be high.
  • Predictors are heteroscedastic in nature.
  • Some predictors are heavily serially correlated, many are not.
  • Predictors all evolve continuously and are decimal processes. They are all bell shaped.
  • Some could be non-stationary, but are thought to carry critical information about the evolution of the response process that would be lost upon differencing or similar $I(0)$ transforms.
  • My forecasting model will be linear, but least squares loss function is not certain (perpendicular regression and quantile regression and discrete response regression will be tried).
  • Standard error unbiasedness and variance is not a direct consideration. All I care about is out of sample forecast accuracy and model generalisation.

I know I can use anything from non-negative garotte, ridge, lasso, elastic nets, random subspace learning, PCA/manifold learning, least angle regression and various yucky hacks to select based on out of sample forecast performance. But is there anything specific to forecasting, or to the characteristics of my data, that would push me one way over another (time efficiency is important - trying everything and selecting the best is not practical).

Note that decision boundaries on the forecasts are not intended to be part of the regularisation algorithm, but integration could be possible.

I will also be using a supercomputer.

  • $\begingroup$ Just want to clarify the second point. By saying that change in response happens 2% of the time, you mean that your time series which you want to forecast contains only 2% non-zero values? $\endgroup$
    – mpiktas
    Oct 3 '13 at 14:18
  • $\begingroup$ @mpiktas Please see my update. But your interpretation is correct, except it's 0.3% $\endgroup$ Oct 3 '13 at 14:23

Such type of question is not really suitable for this site, since there can be no one correct answer to such a question. Here are my observations.

  1. Your response variable is censored, so linear model may be not the best model, since clearly it will not produce the exact zeroes. I would look into some sort of censored regression, like tobit regression.

  2. Since you want a regression model, forecasting the future values of the response variable involves forecasting the future values of the predictor variables. So when evaluating forecasting performance you should use the forecasts of predictor variables, not their out-of-sample values to get more reliable forecasting performance estimate.

  • $\begingroup$ I don't understand the second point. I have some function of the features $f(X(t)) = Y(t+1)$ that I need to estimate. Once I have $\hat{f}$ then I only need information at time $t$ to get my forecast -- no future information needed. $\endgroup$ Oct 4 '13 at 9:56
  • $\begingroup$ Yes, but in the usual regression it is assumed that $Y_t=f(X_t)$. Writing $Y_{t+1}=f(X_t)$ assumes that there is no contemporaneous effect, which might be false. Although for forecasting problem with huge sample size this is assumption can be made. $\endgroup$
    – mpiktas
    Oct 4 '13 at 10:42
  • $\begingroup$ You are indeed correct. Well said. But I am confused about the wording of the sentence in your answer. I do not know what you mean by "So when evaluating forecasting performance you should use the forecasts of predictor variables, not their out-of-sample values to get more reliable forecasting performance estimate." Where do the forecasts of these predictors come from? In everything I've read about prediction, to evaluate performance you simply look at things like MAE, MAPE, etc, on the response forecasts vs. the actual response values. Predictor forecasts don't come into it. $\endgroup$ Oct 4 '13 at 10:46
  • 1
    $\begingroup$ If you use the model $Y_{t+1}=f(X_t)$ and are interested in one step ahead forecasts then it is ok. More common situation is the model $Y_t=f(X_t)$. When you divide into in-sample and out-of-sample and measure the forecasting performance you can use $X_t$ from the out-of-sample to get $\hat{Y_t}$ and then look at MAE, MAPE, etc. But that would not be right, since in real life exercise $X_t$ are unavailable. Hence although you have out-of-sample $X_t$ you need to forecast them and use only out-of-sample $Y_t$ to measure forecasting performance. $\endgroup$
    – mpiktas
    Oct 4 '13 at 11:21

You could try input variable selection based on partial mutual information. Paper talks about algorithm details.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.