1
$\begingroup$

*********Okay so I figured out what was wrong! I wasn't centering the date like the lm.ridge function does. However I still cannot reproduce the intercept that lm.ridge gives me.

According to my research you can simulate a ridge regression by adding "phony data" to the end of a normal OLS regression. One of many places that corroborate this notion is this CV thread: Phoney data and ridge regression are the same?

However I fail to replicate the results in R. Here are my three variables:

> test_0
12    34    24    64   746    24    23    42     7     8     3     4    45   675     3     4    34    43  56   674     3     4    54    34    23    34   435    56    56   234   657    89   980     8    76    65 45564    67    76   789

> test_1
34    24    64   746    24    23    42     7     8     3     4    45   675     3     4    34    43    56 674     3     4    54    34    23    34   435    56    56   234   657    89   980     8    76    65 45564  67    76   789     6

> test_2
24    64   746    24    23    42     7     8     3     4    45   675     3     4    34    43    56   674 3     4    54    34    23    34   435    56    56   234  657    89   980     8    76    65 45564    67 76   789     6     5

I then append 2 new rows (for the number of independent vars). To test_0 I append two zeros. To test_1 I append a sqrt(.5) and 0. To test_2 I append a 0 and sqrt(.5)

> a = c(test_0, 0, 0)
> b = c(test_1, (sqrt(.5)), 0)
> c = c(test_2, 0, (sqrt(.5)))

Then I run two models, lm and lm.ridge:

>reg = lm(a~b+c)
>ridge = lm.ridge(test_0~test_1+test_2, lambda=.5)
> reg
 Call:
 lm(formula = a ~ b + c)

 Coefficients:
 (Intercept)            b            c  
  1305.42310     -0.02926     -0.02862  

> ridge
                      test_1        test_2 
 1374.16801379   -0.03059968   -0.02996396 

The coefficients are different but they should be the same. Why is this the case?

I have also tried the above using a lambda of 1 and still get the inconsistency.

$\endgroup$
  • $\begingroup$ Okay so I figured out what was wrong! I wasn't centering the date like the lm.ridge function does. However I still cannot reproduce the intercept that lm.ridge gives me. $\endgroup$ – Stevens Sep 18 '15 at 18:11
  • 1
    $\begingroup$ You can post this as an answer to help future users. They may even upvote your answer to allow you to accrue reputation. $\endgroup$ – Sycorax Sep 18 '15 at 19:48
  • $\begingroup$ You should mention in your post what package lm.ridge is from $\endgroup$ – Glen_b Sep 19 '15 at 0:18
2
$\begingroup$

The reason my values are off was because the the regular ridge regression method assumes you centralize (standardize) your data in the y vector and X matrix.

If you centralize and then run an OLS with the "phoney data" added in, you get the Ridge Regression betas. The intercept is another story, which I haven't figured out.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.