I am trying to fit a problem with regsubsets with leaps in R. My problem is particularly strongly collinear, which is why I chose to use it in the first place.

The number of variables is about 200 and I have about 2 million independent observations. All the variables have a strong correlation structure with each other.

On running regsubsets with really.big = TRUE, and nvmax = 5 and nbest = 1, I get the following:

Error in leaps.setup - 31 linear dependencies found

and it crashes. All I am looking to do is a simple forward stepwise, say order the variables in the order of correlation and run nested regressions.

Is that too much for the software to handle? I think the problem is well posed in that sense.

  • $\begingroup$ I'm getting the same error you are, and it's driving me crazy. I think it has something to do with the number of vars with respecto to the number of observations but I'm not sure. Any suggestions would be greatly appreciated. $\endgroup$ – Ainhoa Nov 26 '12 at 17:19

Did you instruct regsubsets to do a forward selection? The default is "exhaustive", I believe.

In any case, the collinearities will still cause trouble. Any time regsubsets considers a collection of variables that are too collinear (i.e. the design matrix is practically singular), it will fail.

"Best subset" methods can be unstable with multiple regression, especially when there are a lot of variables. You might want to try a random forest approach.

  • $\begingroup$ I did that, and it still doesnt work. As I said before what is wrong is just framing the algorithm as simple as 1) Pick the variables in order of correlation 2) Regress the subsequent residuals. Is this not what forward stepwise is ? $\endgroup$ – gbh. Nov 25 '12 at 19:48
  • 2
    $\begingroup$ Almost, but not quite. It's true that you build up the model by adding the "best" variable of the remainder to those that you already have. Once a variable goes into the model, it remains there. However the addition of each new variable involves inverting the X'X - where X is the design matrix at that stage. When the columns of X become linearly dependent (or nearly so), then inversion of that matrix will cause the algorithm to fail. $\endgroup$ – Placidia Nov 26 '12 at 21:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.