2
$\begingroup$

How do you take the derivative of the function

$$s(\beta)=\displaystyle\sum\frac{Y_i}{X_i^\intercal\beta}X_i-\sum\frac{1-Y_i}{1-X_i^\intercal\beta}X_i?$$

Attempt:

$$H(\beta)=\frac\partial{\partial\beta}s(\beta)=\frac \partial {\partial \beta} \sum\left(\frac{Y_i}{X_i^\intercal\beta}X_i-\frac{1-Y_i}{1-X_i^\intercal\beta}X_i\right)=\\[2em]\sum\left(-\frac{Y_i}{(X_i^\intercal\beta)^2}X_iX_i^\intercal-\frac{1-Y_i}{(1-X_i^\intercal\beta)^2}X_iX_i^\intercal\right)$$

as well as calculate $EH(\beta)$?

I need this because I'm trying to calculate asymptotic variance of a maximum likelihood estimator $-I(\beta)^{-1}=-E(H(\beta))^{-1}$.

By the way, $E(Y_i|X_i)=X_i^\intercal\beta$ here if you want to take the solution all the way.

$\endgroup$
14
  • $\begingroup$ Is $\beta$ a vector or a single coefficient? Based on the expression $X_i^T\beta$ (which is itselve also unclear), which looks like a vector product, I would suspect it is a vector. Could you clarify this. $\endgroup$ Jan 21, 2021 at 6:21
  • $\begingroup$ $X_i,\beta$ are vectors of length P $\endgroup$
    – user308286
    Jan 21, 2021 at 6:34
  • $\begingroup$ while $Y_i\sim Bern(\pi_i)\in \{0,1\}$ and $\pi_i=X_i^\intercal\beta$, the identity link $\endgroup$
    – user308286
    Jan 21, 2021 at 6:49
  • $\begingroup$ I took the expectation of the hessian and came up with an answer. It’s all good $\endgroup$
    – user308286
    Jan 21, 2021 at 7:13
  • 1
    $\begingroup$ Is $s(\beta)$ a vector or a scalar? I imagine that $\frac{Y_i}{X_i^T\beta}$ is a scalar $\frac{Y_i}{\hat{Y}_i}$ (and by the way: $X_i^T\beta = \hat{Y}_i$ but not $X_i^T\beta = E(Y_i|X_i)$. Isn't it an estimate of the expectation and not equal to the expectation?). Then $\frac{Y_i}{X_i^T\beta} X_i$ is a vector. But... the Hessian is a derivative of a scalar valued function. $\endgroup$ Jan 21, 2021 at 9:13

1 Answer 1

1
$\begingroup$

It seems like you are computing the Hessian of a logarithm of the likelihood $\mathcal{L} = \prod p_i^{Y_i} (1-p_i)^{1-Y_i}$ with $p_i = X_i \cdot \beta$.

For simplicity you might consider instead to compute the Hessian of a single observation $i$ because that subscript $_i$ seems to complicate the computation. $\mathcal{l} = p^Y (1-p)^{1-Y}$ with $p = X \cdot \beta$

$$\begin{array}{rcl}\frac{\partial^2}{\partial \beta_i \partial \beta_j} \log {p}^Y (1-p)^{1-Y} &=& \frac{\partial^2}{\partial \beta_i \partial \beta_j} \left( Y \log( \boldsymbol{X} \cdot \boldsymbol\beta )+ (1-Y) \log (1-\boldsymbol{X} \cdot \boldsymbol\beta) \right)\\ &=& \frac{\partial}{\partial \beta_j} YX_i \frac{1}{(\boldsymbol{X} \cdot \boldsymbol\beta)} + (1-Y)X_i \frac{1}{ (1-\boldsymbol{X} \cdot \boldsymbol\beta)} \\ &=& - YX_i X_j\frac{1}{(\boldsymbol{X} \cdot \boldsymbol\beta)^2} - (1-Y)X_iX_j \frac{1}{ (1-\boldsymbol{X} \cdot \boldsymbol\beta)^2} \\ &=& -X_iX_j \left( \frac{Y}{(\boldsymbol{X} \cdot \boldsymbol\beta)^2} + \frac{1-Y}{(1-\boldsymbol{X} \cdot \boldsymbol\beta)^2} \right) \\ \end{array}$$

And for the Hessian of the likelihood of multiple observations you'd get

$$\begin{array}{rcl}\frac{\partial^2}{\partial \beta_i \partial \beta_j}\log \prod_{\forall k} {p}^{Y_k} (1-p_k)^{1-Y_k} &=& \sum_{\forall{k}} X_{ik}X_{jk} \left( \frac{Y_k}{(\boldsymbol{X_k} \cdot \boldsymbol\beta)^2} + \frac{1-Y_k}{(1-\boldsymbol{X_k} \cdot \boldsymbol\beta)^2} \right) \\ \end{array}$$

Your computation seems to go wrong where you are repeating the index $i$ in a term like $\frac{Y_i}{X_i^\intercal\beta}X_i$ but the $i$ relates to different indexes and your $X$ is actually a matrix with two indices, one indice for the observation number and one indice for the column/coefficient/parameter. The $X_i^T$ and $X_i$ are not just different in that the one is the transpose of the other, it should be $\frac{Y_i}{X_i^\intercal\beta}X_{ij}$ where $X$ is a matrix $X_i$ is a vector (relating to the $i$-th observation) and $X_{ij}$ is a scalar (relating to the ($j$-th component of $X$ for the $i$-th observation).


To compute the expectation... you have $Y=1$ and $H(l)= X_iX_j \frac{1}{(\boldsymbol{X} \cdot \boldsymbol\beta)^2}$ with probability $\boldsymbol{X} \cdot \boldsymbol\beta$ and you $Y=0$ and $H(l)= X_iX_j \frac{1}{(1-\boldsymbol{X} \cdot \boldsymbol\beta)^2}$ with probability $1-\boldsymbol{X} \cdot \boldsymbol\beta$. This makes

$$E(H(l)|\boldsymbol{X},\boldsymbol{\beta}) = -X_iX_j \left( \frac{1}{(\boldsymbol{X} \cdot \boldsymbol\beta)} + \frac{1}{(1-\boldsymbol{X} \cdot \boldsymbol\beta)} \right) = -\frac{X_iX_j }{(\boldsymbol{X} \cdot \boldsymbol\beta)(1-\boldsymbol{X} \cdot \boldsymbol\beta)} = -\frac{X_iX_j }{\text{Var}(Y|\boldsymbol{X},\boldsymbol{\beta})} $$

$\endgroup$
1
  • 1
    $\begingroup$ Thanks for putting together the answer! :-) (I think there’s one little error where you wrote $\prod\log$ for the hessian of multiple observations, should be $\log\prod$ or $\sum\log$.) $\endgroup$
    – user308286
    Jan 22, 2021 at 1:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.