I've read that ridge regression could be achieved by simply adding rows of data to the original data matrix, where each row is constructed using 0 for the dependent variables and the square root of $k$ or zero for the independent variables. One extra row is then added for each independent variable.

I was wondering whether it is possible to derive a proof for all cases, including for logistic regression.

up vote 10 down vote accepted

Ridge regression minimizes $\sum_{i=1}^n (y_i-x_i^T\beta)^2+\lambda\sum_{j=1}^p\beta_j^2$.

(Often a constant is required, but not shrunken. In that case it is included in the $\beta$ and predictors -- but if you don't want to shrink it, you don't have a corresponding row for the pseudo observation. Or if you do want to shrink it, you do have a row for it. I'll write it as if it's not counted in the $p$, and not shrunken, as it's the more complicated case. The other case is a trivial change from this.)

We can write the second term as $p$ pseudo-observations if we can write each "y" and each of the corresponding $(p+1)$-vectors "x" such that

$(y_{n+j}-x_{n+j}^T\beta)^2=\lambda\beta_j^2\,,\quad j=1,\ldots,p$

But by inspection, simply let $y_{n+j}=0$, let $x_{n+j,j}=\sqrt{\lambda}$ and let all other $x_{n+j,k}=0$ (including $x_{n+j,0}=0$ typically).

Then

$(y_{n+j}-[x_{n+j,0}\beta_0+x_{n+j,1}\beta_1+x_{n+j,2}\beta_2+...+x_{n+j,p}\beta_p])^2=\lambda\beta_j^2$.

This works for linear regression. It doesn't work for logistic regression, because ordinary logistic regression doesn't minimize a sum of squared residuals.

[Ridge regression isn't the only thing that can be done via such pseudo-observation tricks -- they come up in a number of other contexts]

  • Thanks, I was already struggling with rewriting everything from logistic regression, but I simply couldn't implement the phoney data method. And I don't trust my own abilities sufficiently to be able to say that it is impossible. – Snowflake Feb 10 '15 at 12:28
  • At least I don't think it is. I'll take another look at the likelihood function. – Glen_b Feb 10 '15 at 12:31
  • 3
    +1 Additional related regression tricks are introduced in answers at stats.stackexchange.com/a/32753 and stats.stackexchange.com/a/26187, inter alia. – whuber Feb 10 '15 at 14:15

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.