2
$\begingroup$

I compute ridge regression results with Matlab, not using their implementation but simply computing (trans(X)X)+kI)^-1+trans(X)y as seen here. The given formula equals the ridge formula given in the link, just above "examples". "trans" stands for "transposed", "^" stands for "taken to the power of". "k" is equivalent to what is usually referred to as lambda.

Ridge regression penalizes for moving away from 0, but I want to penalize for moving away from a certain prior. The prior is different for each coefficient that has to be computed.

Is there a simple way modify the formula above to take the different priors into account?

Thanks

$\endgroup$
  • $\begingroup$ Can you clarify/edit your post? In particular, the equation you've given is not consistent with a ridge regression computation, nor does it look like MATLAB code. $\endgroup$ – cardinal Oct 6 '11 at 17:01
  • $\begingroup$ @Susie Please register your account so you would be able to edit your posts. Just visit stats.stackexchange.com/users/login $\endgroup$ – user88 Oct 6 '11 at 18:04
  • $\begingroup$ @cardinal The posted expression looks fine to me (now, maybe it has been edited since your comment), and agrees with what I get below, where X is replaced by A, k replaced by lambda. $\endgroup$ – DavidR Oct 6 '11 at 19:24
  • 2
    $\begingroup$ @DavidR: There appears to be an extraneous plus sign in there as the dimensions of the two terms don't even match. :) $\endgroup$ – cardinal Oct 6 '11 at 20:02
  • 1
    $\begingroup$ Ok! You are correct. That + should be a multiplication. And there's also an extra paren before the +kI. $\endgroup$ – DavidR Oct 6 '11 at 20:15
3
$\begingroup$

Not quite clear on your notation, so I'll set up the problem from scratch:

We want to map vectors $x\in\mathcal{R}^{m}$ to $y\in\mathcal{R}$. Let $A$ be the design matrix -- that is, our sample of $x$ vectors are rows of $A$. We want to find $\alpha\in\mathcal{R}^{m}$ such that $y\approx x'\alpha$. In our objective function, rather than penalizing with $\lambda\alpha^{T}\alpha$ (the standard ridge penalty), we will penalize with $\lambda(\alpha-\alpha_{0})^{T}(\alpha-\alpha_{0})$.

Then we get $$ \alpha=(A'A+\lambda I)^{-1}(A'y+\lambda\alpha_{0}) $$ as the optimal solution.

Here's a derivation. I consider the slightly more general case of penalizing by $\lambda\left(\alpha-\alpha_{0}\right)^{T}K\left(\alpha-\alpha_{0}\right)$, for any symmetric positive semi-definite matrix $K$. We want to find $$ \arg \min_{\alpha}||A\alpha-y||^{2}+\lambda\left(\alpha-\alpha_{0}\right)^{T}K\left(\alpha-\alpha_{0}\right) $$

Expanding out the quadratic forms and dropping terms not involving $\alpha$, we find that this is equivalent to $$ \arg\min_{\alpha} [\alpha'(A'A+\lambda K)\alpha-2y'A\alpha-2\lambda\alpha_{0}'K\alpha] $$

The derivative of the objective is $$ \partial_{\alpha}\left[\alpha'(A'A+\lambda K)\alpha-2y'A\alpha-2\lambda\alpha_{0}'K\alpha\right]=2(A'A+\lambda K)\alpha-2A'y-2\lambda K\alpha_{0}. $$ Equating to zero, we get $$ (A'A+\lambda K)\alpha=A'y+\lambda K\alpha_{0} $$ If $K$ is symmetric and [strictly] positive definite (such as the identity matrix in the standard ridge regression case), then $\alpha=(A'A+\lambda K)^{-1}\left(A'y+\lambda K\alpha_{0}\right)$, otherwise, we replace the inverse by a generalized inverse, such as the pseudoinverse (pinv in matlab).

$\endgroup$
  • 1
    $\begingroup$ This derivation was sketched recently at stats.stackexchange.com/questions/15991/…. There's no need to expand the forms: they are easier to differentiate directly. If $K$ is not positive-definite, you're in trouble, because you can make the objective function as negative as you like by choosing $\alpha$ for which $(\alpha-\alpha_0)'K(\alpha-\alpha_0) \lt 0$ and choosing $\lambda$ arbitrarily large and positive: generalized inverses will be of no help. $\endgroup$ – whuber Oct 6 '11 at 20:20
  • $\begingroup$ I was saying spd to indicate strictly positive definite, as opposed to just positive semi-definite (which I require throughout). Yeah, good point about not expanding the quadratic form first. $\endgroup$ – DavidR Oct 6 '11 at 20:33
  • $\begingroup$ Ah, I read "spd" as "symmetric positive definite," corresponding to an earlier phrase in your reply! You don't actually require positive definiteness for the derivation of the final equation; positive definiteness is only required to assure that the critical point corresponds to a global minimum. $\endgroup$ – whuber Oct 6 '11 at 21:30
  • 1
    $\begingroup$ "Symmetric" and "strictly" have different meanings. Here, the symmetry is unnecessary--it's merely a convenience to simplify the equations. (When $K$ is not symmetric you can replace it by the overtly symmetric $K' + K$ without changing the form.) "Strictly" means the value of the form is zero only at the zero vector. $\endgroup$ – whuber Oct 6 '11 at 21:57
  • 1
    $\begingroup$ Ok, sorry for the imprecision. I take spd as a shorthand for symmetric and strictly positive definite (and I read it as "symmetric positive definite"). And psd as short hand for symmetric and positive semi-definite (and I read it as positive semi-definite, with the symmetric being assumed). I thought this was common, but I guess it's just a bad habit. I will include all qualifications in the future! $\endgroup$ – DavidR Oct 6 '11 at 22:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.