# The second-derivative of the log-likelihood function in logistic regression (page 120) model

In textbook The Elements of Statistical Learning, for logistic regression (page 120) model, the log-likelihood function can be written as $$\ell(\beta)=\sum_{i=1}^N(y_i\beta^Tx_i-\log(1+e^{\beta^Tx_i}))$$ This is a $$(p+1)$$ nonlinear equations in $$\beta$$. To solve this problem, we use the Newton-Raphson algorithm, which requires the second-derivative $$\frac{\partial^2 \ell(\beta)}{\partial \beta \partial \beta^T}=-\sum_{i=1}^Nx_ix_i^Tp(x_i;\beta)(1-p(x_i;\beta))$$ where $$p(x_i;\beta)=1-\frac{1}{1+e^{\beta^Tx_i}}$$.

My questions are (1) why the second derivate is not defined by $$\frac{\partial^2 \ell(\beta)}{\partial \beta ^2}?$$

(2) What is the result of $$\frac{\partial \ell(\beta)}{\partial \beta^T}=\frac{e^{\beta^Tx_i}x_i}{(1+e^{\beta^Tx_i})^2}?$$ Is it same as $$\frac{\partial \ell(\beta)}{\partial \beta}?$$

• we need a $nxn$ hessian. The notation used is to indicate that we have a square matrix (aka Hessian)
– sku
Sep 23, 2021 at 7:03
• @sku That is not my question. My question is about the notation. Why we use $\frac{1}{\partial \beta \partial \beta^T}$ rather than $\frac{1}{\partial \beta^2}$? Sep 23, 2021 at 7:24
• This notation makes it explicit that result is a square matrix and not just a scalar.
– sku
Sep 23, 2021 at 7:25
• Is this notation the same as $\nabla ^2$? Sep 23, 2021 at 7:26
• Also, why there is $x_i$ in numerator of $\frac{\partial \ell}{\partial \beta^T}$ but not $x_i^T$? However, there is $x_i^T$ in numerator of $\frac{\partial \ell}{\partial \beta}$? Sep 23, 2021 at 7:27

Maybe you are confused by the difference between univariate and multivariate differentiation. Your first derivative is wrt to a vector $$\boldsymbol{\beta}$$ and therefore is expected to be a vector itself (the collection of all partial derivatives). I.e. $$\frac{\partial \mathcal{l}}{\partial \boldsymbol{\beta}^T}= \left[\frac{\partial \mathcal{l}}{\partial \beta_0},\ldots,\frac{\partial \mathcal{l}}{\partial \beta_p}\right]^T.$$ Now, the second derivative is formed by deriving every element in $$\frac{\partial \mathcal{l}}{\partial \boldsymbol{\beta}^T}$$ wrt to $$\boldsymbol{\beta}$$ again, so we have to form $$p +1$$ derivatives for each of the $$p +1$$ elements in $$\frac{\partial \mathcal{l}}{\partial \boldsymbol{\beta}^T}$$, which results in a $$(p +1) \times (p +1)$$ matrix, the Hessian.
So to answer your first question, you have $$\partial \beta^2$$ in the denominator when you derive wrt to one coefficient (the univariate version) and you have $$\partial \boldsymbol{\beta}^T\boldsymbol{\beta}$$ in the denominator when you derive wrt to a vector (the multivariate version).
Your second question is related to the first question. The result of $$\frac{\partial \mathcal{l}}{\partial \boldsymbol{\beta}^T}$$ is a vector and is therefore not the same as the result of $$\frac{\partial \mathcal{l}}{\partial\beta}$$ which is a scalar.
• why there is $x_i$ in numerator of ∂ℓ/∂β^T but not $x^T_i$? However, there is $x^T_i$ in the numerator of ∂ℓ/∂β? Sep 24, 2021 at 9:08
• Sorry, why $\frac{\partial l}{\partial \beta}$ is not a vector? Sep 28, 2021 at 6:21
• It is not a vector because you do not derive with respect to a vector. The $\partial \beta$ in your denominator is not a vector. Sep 28, 2021 at 9:06
• Sorry, but you said $\frac{\partial l}{\partial \beta^T}$ is a vector? Sep 28, 2021 at 21:50
• Can you also write $\frac{\partial l}{\partial \beta}$ precisely? Thanks. Sep 29, 2021 at 3:49