For given cost function $S(\beta) = (Y - X \beta)^T(Y - X \beta) + \lambda \beta^T \beta$, where $\lambda$ is regularization parameter, the $\beta$ that minimizes the given cost function is $\beta = [X^T X + \lambda]^{-1} X^T Y$.

is it right?


1 Answer 1


This is known as ridge regression in statistics (which is a handy search term). You need an $I$ after the $\lambda$ in your last equation.

Is this work for some subject?

  • $\begingroup$ I am trying to understand the regularization parameter for linear regression. And not sure how to take the derivative $\frac{\partial }{\partial \beta} \lambda \beta^T \beta$. Is it correct that answer should be $\beta = [X^T X + \lambda I]^{-1} X^T Y$ $\endgroup$
    – caspik
    Feb 6, 2013 at 0:54
  • $\begingroup$ Well, that's for formula for $\hat{\beta}$. If you're unfamiliar with derivates with respect to a vector, you can compute the relevant derivative term by term. Note that $\lambda\beta^T\beta$ is scalar. Take the derivative of that with respect to $\beta_i$, and collect the terms up into a vector of the same dimensions as $\beta$, to obtain $2\lambda\beta$. $\endgroup$
    – Glen_b
    Feb 6, 2013 at 10:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.