I've read that ridge regression could be achieved by simply adding rows of data to the original data matrix, where each row is constructed using 0 for the dependent variables and the square root of $k$ or zero for the independent variables. One extra row is then added for each independent variable.

I was wondering whether it is possible to derive a proof for all cases, including for logistic regression.

  • $\begingroup$ Nope, I got it from ncss.com/wp-content/themes/ncss/pdf/Procedures/NCSS/… and it was mentioned briefly on page 335-4 $\endgroup$ – Snowflake Feb 10 '15 at 11:58
  • 1
    $\begingroup$ Sorry about deleting the comment on you there. I decided I was mistaken before I saw your reply and deleted it. $\endgroup$ – Glen_b Feb 10 '15 at 12:11
  • 2
    $\begingroup$ A slight generalization of this problem is asked and answered at stats.stackexchange.com/questions/15991. Because it does not address the logistic regression part of this question, I am not voting to merge the two threads. $\endgroup$ – whuber Feb 10 '15 at 14:13

Ridge regression minimizes $\sum_{i=1}^n (y_i-x_i^T\beta)^2+\lambda\sum_{j=1}^p\beta_j^2$.

(Often a constant is required, but not shrunken. In that case it is included in the $\beta$ and predictors -- but if you don't want to shrink it, you don't have a corresponding row for the pseudo observation. Or if you do want to shrink it, you do have a row for it. I'll write it as if it's not counted in the $p$, and not shrunken, as it's the more complicated case. The other case is a trivial change from this.)

We can write the second term as $p$ pseudo-observations if we can write each "y" and each of the corresponding $(p+1)$-vectors "x" such that

$(y_{n+j}-x_{n+j}^T\beta)^2=\lambda\beta_j^2\,,\quad j=1,\ldots,p$

But by inspection, simply let $y_{n+j}=0$, let $x_{n+j,j}=\sqrt{\lambda}$ and let all other $x_{n+j,k}=0$ (including $x_{n+j,0}=0$ typically).



This works for linear regression. It doesn't work for logistic regression, because ordinary logistic regression doesn't minimize a sum of squared residuals.

[Ridge regression isn't the only thing that can be done via such pseudo-observation tricks -- they come up in a number of other contexts]

  • $\begingroup$ Thanks, I was already struggling with rewriting everything from logistic regression, but I simply couldn't implement the phoney data method. And I don't trust my own abilities sufficiently to be able to say that it is impossible. $\endgroup$ – Snowflake Feb 10 '15 at 12:28
  • $\begingroup$ At least I don't think it is. I'll take another look at the likelihood function. $\endgroup$ – Glen_b Feb 10 '15 at 12:31
  • 3
    $\begingroup$ +1 Additional related regression tricks are introduced in answers at stats.stackexchange.com/a/32753 and stats.stackexchange.com/a/26187, inter alia. $\endgroup$ – whuber Feb 10 '15 at 14:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.