I have a multicollinearity problem in a linear regression model and ridge regression was suggested as a solution. So I have spent quite some time researching different ridge regressors in the literature (there's at least a dozen), however in the course of my research I have found that Principal Components Regression can be considered a special case of ridge regression and so I am including it too. Clearly Partial Least Squares regression is closely linked to Principal Component Regression so it seems it should be added too (although I haven't researched that yet). Also when deriving Ridge regression from the Bayesian viewpoint it became obvious that a Kalman filter approach could be used to implement the Bayesian approach so I'm including that too. The Bayes approach seems fairly natural in my application as I have several years worth of data some in which the collinearity effects are present some where they are not so I can build up a decent prior for the Beta parameters for the cases where little or no MC is present.

I have reviewed various comparison papers and they usually compare the estimators on the basis of their Mean Square Error Performance. Generally in the Ridge v Bayes/Kalman there seems to be no clear winner with the performance depending on the orientation of the betas being estimated to the principal components of the design matrix, the level of multicollinearity and the signal to noise ratio. So it would seem that the best estimator can only be defined in the sense of the best for a given problem.

When should I use lasso vs ridge?

Now through my research I have stumbled into Lasso (and Least Angle regression) and from the answers to the previous thread (see above) I can add "Elastic Net" and "non-negative garrote" (and probably more). I just wonder whether it makes sense to add these to my comparison too? If I'm totally honest it seems that I should. The theory behind the Lasso is pretty similar to that used for the Ridge estimator with the optimization being done in the L1 rather than L2 space.

Ideally I would like to compare all the different classes of shrinkage estimators using the best estimator from each class. Right now I have at least a dozen ridge regressors and no way a priori of knowing which would be best for my problem (given the inconclusive results of the comparison papers I mentioned previously). If I only have to add say the "Lasso" and "Elastic-net" and even the "Non-negative Garrotte" then probably I can do it but if they have as many variations as the Ridge literature has and similar problems identifying which one is likely to be best for a given problem then it seems that the whole thing may become rather unwieldy and I can't cover every approach that's ever been suggested in the literature. So my questions are:

  1. What are the different classes of estimators to deal with Multicollinearity in linear regression: eg. Ridge, PCR, PLS, Lasso...
  2. Apart from Ridge regression do these other classes have one implementation that is generally regarded as best. Any paper I have seen that compares Ridge to other classes of estimators, the authors generally use the basic Ridge regression method which is shown to be one of the poorest performing ridge methods.
  3. Do you have experience in comparing Ridge to these other techniques and which was found to be best?
  4. Is it realistic to compare these different classes all together?

Here is a list of the lasso techniques I have found from a review article by Tibshirani (2011):

Grouped Lasso; Elastic Net Lasso; Fusd Lasso; Adaptive Lasso; Graphical Lasso; Dantzig selector; Near Isotonic Regulation; Matrix Completion; Compressive Sensing; Multivariate Methods;

Now my problem consists of only 4 different X values and around 30 y values. The really high multicollinearity occurs between X variables 3 & 4. Although there can be significant multicollinearity with variable 2 as well. (Variable 1 is a constant).

Most of these techniques are suited for p>>N but this seems directly at odds with the comment from Gary here:

When should I use lasso vs ridge?

That lasso should be used when you have high multicollinearity effects and few variables?

In any case given my problem I think that the Lasso and Elastic Net are most suitable. Can anyone shed any light as to whether any of the remaining techniques may be helpful for my problem?


closed as unclear what you're asking by kjetil b halvorsen, mdewey, Peter Flom May 2 at 13:43

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    $\begingroup$ I tried to resist but could not. Have you read Arthur Goldberger's wonderful chapter on micronumerosity? It's quoted in full in this blog post: davegiles.blogspot.com/2011/09/micronumerosity.html $\endgroup$ – Bill Oct 17 '13 at 19:14
  • $\begingroup$ Your initial comment about resisting commenting seems to suggest that you think my research into ridge and other regularization methods is misguided? I have read about micronumerosity in the course of my research. Unfortunately I can't get any more data for a given day, but I can add data in the bayesian sense by building priors based on previous days observations. Could this be used to address micronumerosity? $\endgroup$ – Baz Oct 18 '13 at 4:03
  • $\begingroup$ No, the point of Goldberger's chapter is that multicollinearity is not a problem to be solved via statistical technique. Just like micronumerosity is not a problem to be solved via statistical technique. The standard errors are big because your data don't reveal the thing you are interested in, not because you analyzed them incorrectly. Of course, you can always make the standard errors smaller by bringing in outside information, but then it's the outside information not the data which are identifying the parameter(s) of interest. $\endgroup$ – Bill Oct 18 '13 at 12:02
  • 2
    $\begingroup$ What do you mean by "different ridge regressors"? "Ridge regression" is a regression with a L2 penalty; there is only one "ridge regression", and no different types, classes, or sorts of it. If you want to compare different regularization techniques, it's fine, but only one of them is called "ridge regression", at least as far as I know. $\endgroup$ – amoeba Jan 18 '15 at 21:49
  • 1
    $\begingroup$ Possible duplicate of Why is multicollinearity not checked in modern statistics/machine learning $\endgroup$ – kjetil b halvorsen May 1 at 11:00