# Parameter estimate for linear regression with regularization

For given cost function $S(\beta) = (Y - X \beta)^T(Y - X \beta) + \lambda \beta^T \beta$, where $\lambda$ is regularization parameter, the $\beta$ that minimizes the given cost function is $\beta = [X^T X + \lambda]^{-1} X^T Y$.

is it right?

This is known as ridge regression in statistics (which is a handy search term). You need an $I$ after the $\lambda$ in your last equation.
• I am trying to understand the regularization parameter for linear regression. And not sure how to take the derivative $\frac{\partial }{\partial \beta} \lambda \beta^T \beta$. Is it correct that answer should be $\beta = [X^T X + \lambda I]^{-1} X^T Y$ – caspik Feb 6 '13 at 0:54
• Well, that's for formula for $\hat{\beta}$. If you're unfamiliar with derivates with respect to a vector, you can compute the relevant derivative term by term. Note that $\lambda\beta^T\beta$ is scalar. Take the derivative of that with respect to $\beta_i$, and collect the terms up into a vector of the same dimensions as $\beta$, to obtain $2\lambda\beta$. – Glen_b Feb 6 '13 at 10:23