1
$\begingroup$

In textbook The Elements of Statistical Learning, for logistic regression (page 120) model, the log-likelihood function can be written as $$\ell(\beta)=\sum_{i=1}^N(y_i\beta^Tx_i-\log(1+e^{\beta^Tx_i}))$$ This is a $(p+1)$ nonlinear equations in $\beta$. To solve this problem, we use the Newton-Raphson algorithm, which requires the second-derivative $$\frac{\partial^2 \ell(\beta)}{\partial \beta \partial \beta^T}=-\sum_{i=1}^Nx_ix_i^Tp(x_i;\beta)(1-p(x_i;\beta))$$ where $p(x_i;\beta)=1-\frac{1}{1+e^{\beta^Tx_i}}$.

My questions are (1) why the second derivate is not defined by $$\frac{\partial^2 \ell(\beta)}{\partial \beta ^2}?$$

(2) What is the result of $$\frac{\partial \ell(\beta)}{\partial \beta^T}=\frac{e^{\beta^Tx_i}x_i}{(1+e^{\beta^Tx_i})^2}?$$ Is it same as $\frac{\partial \ell(\beta)}{\partial \beta}?$

$\endgroup$
5
  • $\begingroup$ we need a $nxn$ hessian. The notation used is to indicate that we have a square matrix (aka Hessian) $\endgroup$
    – sku
    Sep 23, 2021 at 7:03
  • $\begingroup$ @sku That is not my question. My question is about the notation. Why we use $\frac{1}{\partial \beta \partial \beta^T}$ rather than $\frac{1}{\partial \beta^2}$? $\endgroup$
    – Hermi
    Sep 23, 2021 at 7:24
  • $\begingroup$ This notation makes it explicit that result is a square matrix and not just a scalar. $\endgroup$
    – sku
    Sep 23, 2021 at 7:25
  • $\begingroup$ Is this notation the same as $\nabla ^2$? $\endgroup$
    – Hermi
    Sep 23, 2021 at 7:26
  • $\begingroup$ Also, why there is $x_i$ in numerator of $\frac{\partial \ell}{\partial \beta^T}$ but not $x_i^T$? However, there is $x_i^T$ in numerator of $\frac{\partial \ell}{\partial \beta}$? $\endgroup$
    – Hermi
    Sep 23, 2021 at 7:27

1 Answer 1

1
$\begingroup$

Maybe you are confused by the difference between univariate and multivariate differentiation. Your first derivative is wrt to a vector $\boldsymbol{\beta}$ and therefore is expected to be a vector itself (the collection of all partial derivatives). I.e. $$\frac{\partial \mathcal{l}}{\partial \boldsymbol{\beta}^T}= \left[\frac{\partial \mathcal{l}}{\partial \beta_0},\ldots,\frac{\partial \mathcal{l}}{\partial \beta_p}\right]^T.$$ Now, the second derivative is formed by deriving every element in $\frac{\partial \mathcal{l}}{\partial \boldsymbol{\beta}^T}$ wrt to $\boldsymbol{\beta}$ again, so we have to form $p +1$ derivatives for each of the $p +1$ elements in $\frac{\partial \mathcal{l}}{\partial \boldsymbol{\beta}^T}$, which results in a $(p +1) \times (p +1)$ matrix, the Hessian.

So to answer your first question, you have $\partial \beta^2$ in the denominator when you derive wrt to one coefficient (the univariate version) and you have $\partial \boldsymbol{\beta}^T\boldsymbol{\beta}$ in the denominator when you derive wrt to a vector (the multivariate version).

Your second question is related to the first question. The result of $\frac{\partial \mathcal{l}}{\partial \boldsymbol{\beta}^T}$ is a vector and is therefore not the same as the result of $\frac{\partial \mathcal{l}}{\partial\beta}$ which is a scalar.

$\endgroup$
5
  • $\begingroup$ why there is $x_i$ in numerator of ∂ℓ/∂β^T but not $x^T_i$? However, there is $x^T_i$ in the numerator of ∂ℓ/∂β? $\endgroup$
    – Hermi
    Sep 24, 2021 at 9:08
  • $\begingroup$ Sorry, why $\frac{\partial l}{\partial \beta}$ is not a vector? $\endgroup$
    – Hermi
    Sep 28, 2021 at 6:21
  • $\begingroup$ It is not a vector because you do not derive with respect to a vector. The $\partial \beta$ in your denominator is not a vector. $\endgroup$
    – BalaGizeh
    Sep 28, 2021 at 9:06
  • $\begingroup$ Sorry, but you said $\frac{\partial l}{\partial \beta^T}$ is a vector? $\endgroup$
    – Hermi
    Sep 28, 2021 at 21:50
  • $\begingroup$ Can you also write $\frac{\partial l}{\partial \beta}$ precisely? Thanks. $\endgroup$
    – Hermi
    Sep 29, 2021 at 3:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.