I just need a simple explanation of what exactly ridge regression is so I can have a decent intuitive understanding of it. I understand it's about applying some sort of penalty to the regression coefficients... but beyond that I'm a little confused about how it is different from other kinds of regression which implement penalties. In what case should you use ridge regression as opposed to some other kind of regression?
Ridge Regression is a remedial measure taken to alleviate multicollinearity amongst regression predictor variables in a model. Often predictor variables used in a regression are highly correlated. When they are, the regression coefficient of any one variable depend on which other predictor variables are included in the model, and which ones are left out. (So the predictor variable does not reflect any inherent effect of that particular predictor on the response variable, but only a marginal or partial effect, given whatever other correlated predictor variables are included in the model). Ridge regression adds a small bias factor to the variables in order to alleviate this problem. Hope that helps.
The posts above nicely describe ridge regression and its mathematical underpinning. However, they don't address the issue of where ridge regression should be used, compared to other shrinkage methods. It might be so because there are no specific situation where one shrinkage method has been shown to perform better than another. There are many different ways of addressing the issue of multicollinearity among the predictor variables, depending on its source. Ridge regression happens to be one of those methods that addresses the issue of multicollinearity by shrinking (in some cases, shrinking it close to or equal to zero, for large values of the tuning parameter) the coefficient estimates of the highly correlated variables.
Unlike least squares method, ridge regression produces a set of coefficient estimates for different values of the tuning parameter. So, it's advisable to use the results of ridge regession (the set of coefficient estimates) with a model selection technique (such as, cross-validation) to determine the most appropriate model for the given data.