Skip to main content
5 of 9
added 375 characters in body
ADAM
  • 751
  • 4
  • 12

Gradient and Hessian of loss function

I'm trying to clear up the calculation of the gradient and Hessian of a loss function in an article that I am currently reading. The loss function is given by

$$\ell(\beta)=\sum_{i=1}^{N} e^{-y_{i}{{x}}^{\top}_{i} \beta}$$ where $x$, $y$, $\beta$ are vectors of the same length, say $p \times 1$. Now, let $X$ denote the design matrix $X=\left[x_{1},{x}_{1},\cdots,{x}_{N} \right]^{\top}$ and $\beta$ is the coefficient vector and $\eta=X\beta$.

Then the author state that $\dot{\ell}(\beta)$, $\ddot{\ell}(\beta)$, ${\ell}^{'}(\eta)$, $\ell^{''}(\eta)$ be the gradient and Hessian of the loss function with respect to $\beta$ and $\eta$, respectively.

The author did not list what those one look like, and I am trying to obtain them, but all my calculation is pretty off here since I am not sure whether the $\eta$ should be substituted in the loss function first and then take the first and second derivative. Or should I assume that $\eta$ is a function of $X$ and use Chain rules?

Updates: For $\ell(\beta)$ the dimensions for each $i$, $(p \times 1)(1 \times p) (p \times 1) = (p \times 1)$

For $\dot{\ell}(\beta)$ the dimensions for each $i$, $(p \times 1)(1 \times p)(p \times 1)(1 \times p) (p \times 1) = (p \times 1)$

For $\ddot{\ell}(\beta)$ the dimensions for each $i$, $(p \times 1)(1 \times p)(p \times 1)(1 \times p) (p \times 1) = (p \times 1)$

\begin{align*} \ell(\beta)&=\sum_{i=1}^{N} e^{-y_{i}{x}^{\top}_{i} \beta}\\ \dot{\ell}(\beta)&=\frac{\partial \ell(\beta)}{\partial \beta}= -\sum_{i=1}^{N} y_{i}{x}^{\top}_{i} e^{-y_{i}{x}^{\top}_{i} \beta} \, \\ \ddot{\ell}(\beta)&= \frac{\partial^{2} \ell(\beta)}{\partial \beta^{2}}= \sum_{i=1}^{N} ( y_{i}{x}^{\top}_{i})^{2} e^{-y_{i}{x}^{\top}_{i} \beta} \\ \end{align*}

\begin{align*} \ell(\eta)&=\sum_{i=1}^{N} e^{-y_{i}{x}^{\top}_{i} \eta}\\ {\ell}^{'}(\eta)&=\frac{\partial \ell(\eta)}{\partial \eta}= -\sum_{i=1}^{N} y_{i}{x}^{\top}_{i} e^{-y_{i}{x}^{\top}_{i} \eta}=-\sum_{i=1}^{N} y_{i}{x}^{\top}_{i} e^{-y_{i}{x}^{\top}_{i} \eta} \, \\ \ell^{''}(\eta)&= \frac{\partial^{2} \ell(\eta)}{\partial \eta^{2}}= \sum_{i=1}^{N} ( y_{i}{x}^{\top}_{i})^{2} e^{-y_{i}{x}^{\top}_{i} \eta}=\sum_{i=1}^{N} ( y_{i}{x}^{\top}_{i})^{2} e^{-y_{i}{x}^{\top}_{i} \eta} \\ \end{align*}

Thank you in advance!

ADAM
  • 751
  • 4
  • 12