For given cost function $S(\beta) = (Y - X \beta)^T(Y - X \beta) + \lambda \beta^T \beta$, where $\lambda$ is regularization parameter, the $\beta$ that minimizes the given cost function is $\beta = [X^T X + \lambda]^{-1} X^T Y$.

is it right?


This is known as ridge regression in statistics (which is a handy search term). You need an $I$ after the $\lambda$ in your last equation.

Is this work for some subject?

  • $\begingroup$ I am trying to understand the regularization parameter for linear regression. And not sure how to take the derivative $\frac{\partial }{\partial \beta} \lambda \beta^T \beta$. Is it correct that answer should be $\beta = [X^T X + \lambda I]^{-1} X^T Y$ $\endgroup$ – caspik Feb 6 '13 at 0:54
  • $\begingroup$ Well, that's for formula for $\hat{\beta}$. If you're unfamiliar with derivates with respect to a vector, you can compute the relevant derivative term by term. Note that $\lambda\beta^T\beta$ is scalar. Take the derivative of that with respect to $\beta_i$, and collect the terms up into a vector of the same dimensions as $\beta$, to obtain $2\lambda\beta$. $\endgroup$ – Glen_b Feb 6 '13 at 10:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.