# Computing ridge regression with prior different from 0

I compute ridge regression results with Matlab, not using their implementation but simply computing (trans(X)X)+kI)^-1+trans(X)y as seen here. The given formula equals the ridge formula given in the link, just above "examples". "trans" stands for "transposed", "^" stands for "taken to the power of". "k" is equivalent to what is usually referred to as lambda.

Ridge regression penalizes for moving away from 0, but I want to penalize for moving away from a certain prior. The prior is different for each coefficient that has to be computed.

Is there a simple way modify the formula above to take the different priors into account?

Thanks

• Can you clarify/edit your post? In particular, the equation you've given is not consistent with a ridge regression computation, nor does it look like MATLAB code. – cardinal Oct 6 '11 at 17:01
• @Susie Please register your account so you would be able to edit your posts. Just visit stats.stackexchange.com/users/login – user88 Oct 6 '11 at 18:04
• @cardinal The posted expression looks fine to me (now, maybe it has been edited since your comment), and agrees with what I get below, where X is replaced by A, k replaced by lambda. – DavidR Oct 6 '11 at 19:24
• @DavidR: There appears to be an extraneous plus sign in there as the dimensions of the two terms don't even match. :) – cardinal Oct 6 '11 at 20:02
• Ok! You are correct. That + should be a multiplication. And there's also an extra paren before the +kI. – DavidR Oct 6 '11 at 20:15

Not quite clear on your notation, so I'll set up the problem from scratch:

We want to map vectors $x\in\mathcal{R}^{m}$ to $y\in\mathcal{R}$. Let $A$ be the design matrix -- that is, our sample of $x$ vectors are rows of $A$. We want to find $\alpha\in\mathcal{R}^{m}$ such that $y\approx x'\alpha$. In our objective function, rather than penalizing with $\lambda\alpha^{T}\alpha$ (the standard ridge penalty), we will penalize with $\lambda(\alpha-\alpha_{0})^{T}(\alpha-\alpha_{0})$.

Then we get $$\alpha=(A'A+\lambda I)^{-1}(A'y+\lambda\alpha_{0})$$ as the optimal solution.

Here's a derivation. I consider the slightly more general case of penalizing by $\lambda\left(\alpha-\alpha_{0}\right)^{T}K\left(\alpha-\alpha_{0}\right)$, for any symmetric positive semi-definite matrix $K$. We want to find $$\arg \min_{\alpha}||A\alpha-y||^{2}+\lambda\left(\alpha-\alpha_{0}\right)^{T}K\left(\alpha-\alpha_{0}\right)$$

Expanding out the quadratic forms and dropping terms not involving $\alpha$, we find that this is equivalent to $$\arg\min_{\alpha} [\alpha'(A'A+\lambda K)\alpha-2y'A\alpha-2\lambda\alpha_{0}'K\alpha]$$

The derivative of the objective is $$\partial_{\alpha}\left[\alpha'(A'A+\lambda K)\alpha-2y'A\alpha-2\lambda\alpha_{0}'K\alpha\right]=2(A'A+\lambda K)\alpha-2A'y-2\lambda K\alpha_{0}.$$ Equating to zero, we get $$(A'A+\lambda K)\alpha=A'y+\lambda K\alpha_{0}$$ If $K$ is symmetric and [strictly] positive definite (such as the identity matrix in the standard ridge regression case), then $\alpha=(A'A+\lambda K)^{-1}\left(A'y+\lambda K\alpha_{0}\right)$, otherwise, we replace the inverse by a generalized inverse, such as the pseudoinverse (pinv in matlab).

• This derivation was sketched recently at stats.stackexchange.com/questions/15991/…. There's no need to expand the forms: they are easier to differentiate directly. If $K$ is not positive-definite, you're in trouble, because you can make the objective function as negative as you like by choosing $\alpha$ for which $(\alpha-\alpha_0)'K(\alpha-\alpha_0) \lt 0$ and choosing $\lambda$ arbitrarily large and positive: generalized inverses will be of no help. – whuber Oct 6 '11 at 20:20
• I was saying spd to indicate strictly positive definite, as opposed to just positive semi-definite (which I require throughout). Yeah, good point about not expanding the quadratic form first. – DavidR Oct 6 '11 at 20:33
• Ah, I read "spd" as "symmetric positive definite," corresponding to an earlier phrase in your reply! You don't actually require positive definiteness for the derivation of the final equation; positive definiteness is only required to assure that the critical point corresponds to a global minimum. – whuber Oct 6 '11 at 21:30
• "Symmetric" and "strictly" have different meanings. Here, the symmetry is unnecessary--it's merely a convenience to simplify the equations. (When $K$ is not symmetric you can replace it by the overtly symmetric $K' + K$ without changing the form.) "Strictly" means the value of the form is zero only at the zero vector. – whuber Oct 6 '11 at 21:57
• Ok, sorry for the imprecision. I take spd as a shorthand for symmetric and strictly positive definite (and I read it as "symmetric positive definite"). And psd as short hand for symmetric and positive semi-definite (and I read it as positive semi-definite, with the symmetric being assumed). I thought this was common, but I guess it's just a bad habit. I will include all qualifications in the future! – DavidR Oct 6 '11 at 22:09