I have a multicollinearity problem in a linear regression model and ridge regression was suggested as a solution. So I have spent quite some time researching different ridge regressors in the literature (there's at least a dozen), however in the course of my research I have found that Principal Components Regression can be considered a special case of ridge regression and so I am including it too. Clearly Partial Least Squares regression is closely linked to Principal Component Regression so it seems it should be added too (although I haven't researched that yet). Also when deriving Ridge regression from the Bayesian viewpoint it became obvious that a Kalman filter approach could be used to implement the Bayesian approach so I'm including that too. The Bayes approach seems fairly natural in my application as I have several years worth of data some in which the collinearity effects are present some where they are not so I can build up a decent prior for the Beta parameters for the cases where little or no MC is present.
I have reviewed various comparison papers and they usually compare the estimators on the basis of their Mean Square Error Performance. Generally in the Ridge v Bayes/Kalman there seems to be no clear winner with the performance depending on the orientation of the betas being estimated to the principal components of the design matrix, the level of multicollinearity and the signal to noise ratio. So it would seem that the best estimator can only be defined in the sense of the best for a given problem.
Now through my research I have stumbled into Lasso (and Least Angle regression) and from the answers to the previous thread (see above) I can add "Elastic Net" and "non-negative garrote" (and probably more). I just wonder whether it makes sense to add these to my comparison too? If I'm totally honest it seems that I should. The theory behind the Lasso is pretty similar to that used for the Ridge estimator with the optimization being done in the L1 rather than L2 space.
Ideally I would like to compare all the different classes of shrinkage estimators using the best estimator from each class. Right now I have at least a dozen ridge regressors and no way a priori of knowing which would be best for my problem (given the inconclusive results of the comparison papers I mentioned previously). If I only have to add say the "Lasso" and "Elastic-net" and even the "Non-negative Garrotte" then probably I can do it but if they have as many variations as the Ridge literature has and similar problems identifying which one is likely to be best for a given problem then it seems that the whole thing may become rather unwieldy and I can't cover every approach that's ever been suggested in the literature. So my questions are:
- What are the different classes of estimators to deal with Multicollinearity in linear regression: eg. Ridge, PCR, PLS, Lasso...
- Apart from Ridge regression do these other classes have one implementation that is generally regarded as best. Any paper I have seen that compares Ridge to other classes of estimators, the authors generally use the basic Ridge regression method which is shown to be one of the poorest performing ridge methods.
- Do you have experience in comparing Ridge to these other techniques and which was found to be best?
- Is it realistic to compare these different classes all together?
Here is a list of the lasso techniques I have found from a review article by Tibshirani (2011):
Grouped Lasso; Elastic Net Lasso; Fusd Lasso; Adaptive Lasso; Graphical Lasso; Dantzig selector; Near Isotonic Regulation; Matrix Completion; Compressive Sensing; Multivariate Methods;
Now my problem consists of only 4 different X values and around 30 y values. The really high multicollinearity occurs between X variables 3 & 4. Although there can be significant multicollinearity with variable 2 as well. (Variable 1 is a constant).
Most of these techniques are suited for p>>N but this seems directly at odds with the comment from Gary here:
That lasso should be used when you have high multicollinearity effects and few variables?
In any case given my problem I think that the Lasso and Elastic Net are most suitable. Can anyone shed any light as to whether any of the remaining techniques may be helpful for my problem?