If we have a paralyzed loss function of the form of: \begin{align} L(\beta)& =\frac{1}{2}(y-X\beta)^T(y-X\beta)+ \lambda \beta^T f(\beta) \end{align} where $X_{n\times m}$ and $\beta_{m \times 1}$ and $f$ is considered as a column vector. Then what is the derivative of this function with respect to $\beta$. Note that it is derivative with respect to a vector. \begin{align} \frac{\partial L(\beta)}{\partial \beta}& =\frac{\partial}{\partial \beta}\bigg(\frac{1}{2}(y-X\beta)^T(y-X\beta)+ \lambda \beta^T f(\beta)\bigg) \end{align}

Sorry for the silly question. I just confused myself.


1 Answer 1



From (84):

$$ \begin{align} \frac{\partial}{\partial \beta}\bigg((y-X\beta)^T(y-X\beta)\bigg) & = -2 X^T(y-X\beta) \end{align} $$

And using (93):

$$ \begin{align} \frac{\partial}{\partial \beta}\bigg(\lambda\beta^Tf(\beta)\bigg) &= \lambda \bigg( {\bigg[ \frac{\partial \beta}{\partial \beta} \bigg]}^Tf(\beta) + \bigg[ \frac{\partial f(\beta)}{\partial \beta}\bigg]_{m\times m}^T\beta\bigg) \\ &= \lambda\bigg(I_{m \times m}f(\beta)+\bigg[ \frac{\partial f(\beta)}{\partial \beta}\bigg]_{m\times m}^T\beta_{m \times 1}\bigg) \end{align} $$

  • $\begingroup$ This was what I guessed before but the problem is that the first part (in your answer) is a $m\times 1$ vector and the second part is summation of a $m \times 1 $ and $1\times 1$ that is strange! $\endgroup$
    – TPArrow
    Commented Jun 28, 2015 at 22:37
  • 1
    $\begingroup$ Edited. I was using column differentiation for the second part. Note that differentiation of a vector wrt vector is a matrix. $\endgroup$ Commented Jun 28, 2015 at 23:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.