I can understand ridge regression is better than ordinary regression in case of multiple-collinarity.

 x1 <- rnorm(100)
 x2 <- rnorm(100,mean=x1,sd=.01)
 y <- rnorm(100,mean=3+x1+x2)
 myd <- data.frame(x1, x2,y)
          x1        x2         y
x1 1.0000000 0.9999383 0.8701704
x2 0.9999383 1.0000000 0.8702772
y  0.8701704 0.8702772 1.0000000

The ordinary regression:

(Intercept)          x1          x2 
   3.063971   -1.027534    3.097806 

The ridge regression using MASS package

               x1       x2 
3.066382 1.019101 1.043206 

The lambda (shrinkage) = 0 is equal to ordinary regression.

                 x1        x2 
 3.063971 -1.027534  3.097806


(1) How can we show that ridge regression is performing better than ordinary? any estimators like error ? How to calculate ?

(2) Are there other situations where ridge is better OLR ?


I try for an answer, but a rather general one.

(1) It depends on what you mean by "performing better". Often, performance is measured in terms of the capability to generalize and forecast. For this cross-validation is an often used tool, where you repeatedly divide the data into a training and test set, fit the model using the training set, and then take the deviation between forecast and test set as a measure for the generalization capability.

(2) There are many of these situations. The main reason is that ridge regression often can avoid overfitting. A basic example is given at the beginning of Bishop's machine learning book: Here, a polynomial of order nine is fitted to random realizations of a sine curve with added noise. Without ridge regression, the fit obviously seems to overfit for $M=9$:

... well, obviously at least when you additionally see the corresponding sine curve. But even without this information, one should -- according to Ockham's razor -- prefer simple models, in this case the $M=3$ polynomial on the left.

Here are the corresponding results using ridge regression:

You see the benefits, but also the dangers. If you choose $\lambda=1$ (i.e. $\ln \lambda =0$) as is done on the right-hand side, you obtain a fit which most people will find disappointing. For $\ln \lambda = -18$ you retain a simple and obviously appropriate description similar to the $M=3$ polynomial.

The parameter $\lambda$ therefore is seen to reduce the complexity of the model. That is, you can assume a sophisticated model and let the procedure automatically reduce the complexity when it is needed. In general, this is a way to avoid the task of finding an appropriate model specific to each new dataset -- instead, you simply pick a general model and then reduce its complexity until you hopefully get the desired result. Of course, by this you get a further free parameter $\lambda$ which must be properly estimated. Again, this is often done by cross-validation.

Finally, here you see the influence of the ridge parameter on training and test error:

Naturally, with growing $\lambda$, the training error increases as the residual sum of squares becomes larger. At the same time. however, On the other hand, you see that the test-error reaches a minimum somewhere around $\ln \lambda=-30$, which suggests that this is a good value for generalization tasks.

  • $\begingroup$ thanks +1 for the answer, do you mean accuracy of prediction as a measure ? I could find a reference where we can calculate standard error - but do not know how was calculated ? There are other measures I did not figure out - see: web.as.uky.edu/statistics/users/pbreheny/764-F11/notes/9-1.pdf $\endgroup$ – rdorlearn Jul 18 '14 at 13:20
  • $\begingroup$ The basic measure which you probably should try first is the squared error between prediction and target. That is, you do K-fold cross validation and average the error of each test case. You can then optimize this error with respect to $\lambda$ to find the lambda which gives the best predictions. I think this is also what your linked talk is suggesting $\endgroup$ – davidhigh Jul 18 '14 at 23:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.