Computing ridge regression with prior different from 0 I compute ridge regression results with Matlab, not using their implementation but simply computing (trans(X)X)+kI)^-1+trans(X)y as seen here. The given formula equals the ridge formula given in the link, just above "examples". "trans" stands for "transposed", "^" stands for "taken to the power of". "k" is equivalent to what is usually referred to as lambda.
Ridge regression penalizes for moving away from 0, but I want to penalize for moving away from a certain prior. The prior is different for each coefficient that has to be computed.
Is there a simple way modify the formula above to take the different priors into account?
Thanks
 A: Not quite clear on your notation, so I'll set up the problem from
scratch:
We want to map vectors $x\in\mathcal{R}^{m}$ to $y\in\mathcal{R}$. Let $A$
be the design matrix -- that is, our sample of $x$ vectors are rows
of $A$. We want to find $\alpha\in\mathcal{R}^{m}$ such that $y\approx x'\alpha$.
In our objective function, rather than penalizing with $\lambda\alpha^{T}\alpha$ (the standard ridge penalty),
we will penalize with $\lambda(\alpha-\alpha_{0})^{T}(\alpha-\alpha_{0})$.
Then we get
$$
\alpha=(A'A+\lambda I)^{-1}(A'y+\lambda\alpha_{0})
$$
as the optimal solution. 
Here's a derivation. I consider the slightly more general case of penalizing by $\lambda\left(\alpha-\alpha_{0}\right)^{T}K\left(\alpha-\alpha_{0}\right)$,
for any symmetric positive semi-definite matrix $K$. We want to find
$$
\arg \min_{\alpha}||A\alpha-y||^{2}+\lambda\left(\alpha-\alpha_{0}\right)^{T}K\left(\alpha-\alpha_{0}\right)
$$
Expanding out the quadratic forms and dropping terms not involving $\alpha$, we find that this is equivalent to 
$$
 \arg\min_{\alpha} [\alpha'(A'A+\lambda K)\alpha-2y'A\alpha-2\lambda\alpha_{0}'K\alpha]
$$
The derivative of the objective is
$$
\partial_{\alpha}\left[\alpha'(A'A+\lambda K)\alpha-2y'A\alpha-2\lambda\alpha_{0}'K\alpha\right]=2(A'A+\lambda K)\alpha-2A'y-2\lambda K\alpha_{0}.
$$
Equating to zero, we get
$$
(A'A+\lambda K)\alpha=A'y+\lambda K\alpha_{0}
$$
 If $K$ is symmetric and [strictly] positive definite (such as the identity matrix in the standard ridge regression case), then $\alpha=(A'A+\lambda K)^{-1}\left(A'y+\lambda K\alpha_{0}\right)$,
otherwise, we replace the inverse by a generalized inverse, such as
the pseudoinverse (pinv in matlab).
