I'm very impressed by this plot: Why does ridge estimate become better than OLS by adding a constant to the diagonal?

Does someone has any clue about how to plot this on R? I mean, how to get RSS values for differents parameter estimation? With lm function, I will have only one solution!

Thanks for your help!


It's a plot of the objective function (i.e. the sum of squares, the thing you're trying to minimize with OLS) as a function of the coefficients. One way to compute this is to generate a "grid" of possible coefficients and then compute the sum of squares at each point on the grid.

That is, we fit a regression by finding $\hat\beta \equiv \arg \min_\beta \{ y-x\beta \}$, i.e. the $\hat\beta$ that minimizes $SS_{reg}$. But we can compute $y-x\beta$ for any $\beta$ value. So we can generate a bunch of different $\beta$ values and just plug each one into that $SS_{reg}$ formula. In this case, $\beta$ has two components (i.e. the intercept and slope), so you can visualize all of the possible $SS_{reg}$s as a 3D surface.

As for generating the plot itself, look here for a complete example with R code.

  • $\begingroup$ Thanks for this answer, very clear! But I have a concern, how do I generate the different value of β? I cannot use the lm function, can I? $\endgroup$ – PLOTZ Oct 17 '14 at 6:22
  • 1
    $\begingroup$ PLOTZ - you choose a grid of values for the betas and evaluate $\text{SS}_\text{reg}$ at each grid point. Which values are on your grid depends on which parts of the function you want to look at, but to include the minimum it should include the least squares estimate, probably near the center of the grid. IIRC I used outer for this, but expand.grid would be another good choice. $\endgroup$ – Glen_b Oct 17 '14 at 9:59
  • 1
    $\begingroup$ In order to make the sum of squares have the long valley, I had the data in $(x_1,x_2)$ space lie scattered about a straight line. Because many planes pass through the line, there's not much information in the data about which plane it should be (hence the valley - there's many combinations of $(\hat{\beta_1},\hat{\beta_2})$ with nearly the same fit). $\endgroup$ – Glen_b Oct 17 '14 at 10:49
  • $\begingroup$ Thank you Glen. Let me try this, and maybe I'll get back to you if I am stuck. I am just à bit disapointed because if I understand well, thetas do not come from a real solving of regression vs. Ridge regression. Because ideally, I would like to do the same kind of plot for the lasso (l1 norm), and I don't how the thetas should look like ! $\endgroup$ – PLOTZ Oct 17 '14 at 21:06
  • $\begingroup$ @PLOTZ it would be trivial to make a plot for LASSO instead of Ridge. What thetas are you talking about? $\endgroup$ – shadowtalker Oct 17 '14 at 21:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.