3
$\begingroup$

I have a multicollinearity problem in a linear regression model and ridge regression was suggested as a solution. So I have spent quite some time researching different ridge regressors in the literature (there's at least a dozen), however in the course of my research I have found that Principal Components Regression can be considered a special case of ridge regression and so I am including it too. Clearly Partial Least Squares regression is closely linked to Principal Component Regression so it seems it should be added too (although I haven't researched that yet). Also when deriving Ridge regression from the Bayesian viewpoint it became obvious that a Kalman filter approach could be used to implement the Bayesian approach so I'm including that too. The Bayes approach seems fairly natural in my application as I have several years worth of data some in which the collinearity effects are present some where they are not so I can build up a decent prior for the Beta parameters for the cases where little or no MC is present.

I have reviewed various comparison papers and they usually compare the estimators on the basis of their Mean Square Error Performance. Generally in the Ridge v Bayes/Kalman there seems to be no clear winner with the performance depending on the orientation of the betas being estimated to the principal components of the design matrix, the level of multicollinearity and the signal to noise ratio. So it would seem that the best estimator can only be defined in the sense of the best for a given problem.

When should I use lasso vs ridge?

Now through my research I have stumbled into Lasso (and Least Angle regression) and from the answers to the previous thread (see above) I can add "Elastic Net" and "non-negative garrote" (and probably more). I just wonder whether it makes sense to add these to my comparison too? If I'm totally honest it seems that I should. The theory behind the Lasso is pretty similar to that used for the Ridge estimator with the optimization being done in the L1 rather than L2 space.

Ideally I would like to compare all the different classes of shrinkage estimators using the best estimator from each class. Right now I have at least a dozen ridge regressors and no way a priori of knowing which would be best for my problem (given the inconclusive results of the comparison papers I mentioned previously). If I only have to add say the "Lasso" and "Elastic-net" and even the "Non-negative Garrotte" then probably I can do it but if they have as many variations as the Ridge literature has and similar problems identifying which one is likely to be best for a given problem then it seems that the whole thing may become rather unwieldy and I can't cover every approach that's ever been suggested in the literature. So my questions are:

  1. What are the different classes of estimators to deal with Multicollinearity in linear regression: eg. Ridge, PCR, PLS, Lasso...
  2. Apart from Ridge regression do these other classes have one implementation that is generally regarded as best. Any paper I have seen that compares Ridge to other classes of estimators, the authors generally use the basic Ridge regression method which is shown to be one of the poorest performing ridge methods.
  3. Do you have experience in comparing Ridge to these other techniques and which was found to be best?
  4. Is it realistic to compare these different classes all together?

Here is a list of the lasso techniques I have found from a review article by Tibshirani (2011):

Grouped Lasso; Elastic Net Lasso; Fusd Lasso; Adaptive Lasso; Graphical Lasso; Dantzig selector; Near Isotonic Regulation; Matrix Completion; Compressive Sensing; Multivariate Methods;

Now my problem consists of only 4 different X values and around 30 y values. The really high multicollinearity occurs between X variables 3 & 4. Although there can be significant multicollinearity with variable 2 as well. (Variable 1 is a constant).

Most of these techniques are suited for p>>N but this seems directly at odds with the comment from Gary here:

When should I use lasso vs ridge?

That lasso should be used when you have high multicollinearity effects and few variables?

In any case given my problem I think that the Lasso and Elastic Net are most suitable. Can anyone shed any light as to whether any of the remaining techniques may be helpful for my problem?

$\endgroup$
8
  • 1
    $\begingroup$ I tried to resist but could not. Have you read Arthur Goldberger's wonderful chapter on micronumerosity? It's quoted in full in this blog post: davegiles.blogspot.com/2011/09/micronumerosity.html $\endgroup$
    – Bill
    Commented Oct 17, 2013 at 19:14
  • $\begingroup$ Your initial comment about resisting commenting seems to suggest that you think my research into ridge and other regularization methods is misguided? I have read about micronumerosity in the course of my research. Unfortunately I can't get any more data for a given day, but I can add data in the bayesian sense by building priors based on previous days observations. Could this be used to address micronumerosity? $\endgroup$
    – Baz
    Commented Oct 18, 2013 at 4:03
  • 1
    $\begingroup$ To use an example I always use in class, suppose you have a sample of shoe factories with variables for total production cost, number of right shoes, number of left shoes. You can get excellent, plausible estimates of the extra cost from producing an extra pair of shoes and excellent forecasts of how much a shoe factory producing 10000 pairs of shoes will cost to run. But, your estimates of how much it would cost to produce 1000 left shoes and 6000 right shoes will be terrible, because nothing like that ever happens in the data. $\endgroup$
    – Bill
    Commented Oct 18, 2013 at 17:10
  • 2
    $\begingroup$ What do you mean by "different ridge regressors"? "Ridge regression" is a regression with a L2 penalty; there is only one "ridge regression", and no different types, classes, or sorts of it. If you want to compare different regularization techniques, it's fine, but only one of them is called "ridge regression", at least as far as I know. $\endgroup$
    – amoeba
    Commented Jan 18, 2015 at 21:49
  • 1
    $\begingroup$ Possible duplicate of Why is multicollinearity not checked in modern statistics/machine learning $\endgroup$ Commented May 1, 2019 at 11:00

0