4
$\begingroup$

I have the following regression problem

I have about 60 independent variables; some of them have a high correlation with others. I have around 3 million observations

(1) - My main goal is out-of-sample-prediction, so my main question is: which regularization method should I use in this case?

Some more questions (assumptions I have, probably a little confused)

(2) - Ridge regression, while not completely removing coefficients, would keep those coefficients low that lasso/elastic net/BIC would completely remove; is that correct? (If it doesn't, would that be a problem?)

(3) - If I wanted to use AIC/BIC in this case, I would have to test all possible combinations of the 60 independant variables?

(4) - Would it make sense to start with AIC/BIC, then later do ridge regression with the remaining independant variables? (I guess ridge regression after AIC/BIC might make sense because some of the independant variables correlate with others?)

Thanks

|cite|improve this question
$\endgroup$
  • 4
    $\begingroup$ With 3 million observations, and 60 features, there is no ostensible reason to perform variable selection, unless you are running into numerical problems. Particularly if you are interested in prediction. $\endgroup$ – tchakravarty Nov 30 '12 at 11:24
  • $\begingroup$ You don't specify why you need parsimony in your model. Is it difficult or expensive to collect all the 60 variables? Just like @fgnu asked, are you running into numerical problems? $\endgroup$ – user765195 Nov 30 '12 at 19:06
3
$\begingroup$

There seem to be two related issues here: 1) Overfitting and 2) Collinearity. As @fg said in a comment, with so many observations overfitting is not likely to be a real problem. However, collinearity may be.

High correlations among IVs is often a sign of problematic collinearity - that is, collinearity that can cause the model to be poorly estimated - but it is neither a necessary nor a sufficient condition for that. Since you are estimating a linear model I suggest getting the condition indices and proportion of variance explained matrix (you don't say if you are using R, SAS, SPSS or what, but this is available in all three of those and probably others). High condition indices (30 or so is a recommended threshold) that have associated high proportion of variance explained on two or more variables may cause problems.

An alternative to this (which also works well for nonlinear models), if you are using R, is the perturb package.

The main problem caused is that small changes in the input can cause large changes in the model (and, thus, possibly large changes in predictions).

|cite|improve this answer
$\endgroup$
  • $\begingroup$ What about point 3. in here (Rob J. Hyndman's blog): "Mul­ti­collinear­ity has a very dif­fer­ent impact if your goal is pre­dic­tion from when your goal is esti­ma­tion. When pre­dict­ing, mul­ti­collinear­ity is not really a prob­lem pro­vided the val­ues of your pre­dic­tors lie within the hyper-​​region of the pre­dic­tors used when esti­mat­ing the model." In the question, it is stated that the goal is prediction; so maybe multicollinearity is not a problem? $\endgroup$ – Richard Hardy Nov 26 '14 at 8:51
  • $\begingroup$ I think there could still be a problem because 1) The post says the main goal is prediction, which sort of implies that he's also interested in explanation 2) He wants to predict out of sample, which could get out of the hyper-region. $\endgroup$ – Peter Flom♦ Nov 26 '14 at 12:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.