$\frac\partial{\partial\beta}\left(\sum\frac{Y_i}{X_i^\intercal\beta}X_i-\sum\frac{1-Y_i}{1-X_i^\intercal\beta}X_i\right)$

Question

How do you take the derivative of the function

$$s(\beta)=\displaystyle\sum\frac{Y_i}{X_i^\intercal\beta}X_i-\sum\frac{1-Y_i}{1-X_i^\intercal\beta}X_i?$$

Attempt:

$$H(\beta)=\frac\partial{\partial\beta}s(\beta)=\frac \partial {\partial \beta} \sum\left(\frac{Y_i}{X_i^\intercal\beta}X_i-\frac{1-Y_i}{1-X_i^\intercal\beta}X_i\right)=\\[2em]\sum\left(-\frac{Y_i}{(X_i^\intercal\beta)^2}X_iX_i^\intercal-\frac{1-Y_i}{(1-X_i^\intercal\beta)^2}X_iX_i^\intercal\right)$$

as well as calculate $EH(\beta)$?

I need this because I'm trying to calculate asymptotic variance of a maximum likelihood estimator $-I(\beta)^{-1}=-E(H(\beta))^{-1}$.

By the way, $E(Y_i|X_i)=X_i^\intercal\beta$ here if you want to take the solution all the way.

Answer 1

It seems like you are computing the Hessian of a logarithm of the likelihood $\mathcal{L} = \prod p_i^{Y_i} (1-p_i)^{1-Y_i}$ with $p_i = X_i \cdot \beta$.

For simplicity you might consider instead to compute the Hessian of a single observation $i$ because that subscript $_i$ seems to complicate the computation. $\mathcal{l} = p^Y (1-p)^{1-Y}$ with $p = X \cdot \beta$

$$\begin{array}{rcl}\frac{\partial^2}{\partial \beta_i \partial \beta_j} \log {p}^Y (1-p)^{1-Y} &=& \frac{\partial^2}{\partial \beta_i \partial \beta_j} \left( Y \log( \boldsymbol{X} \cdot \boldsymbol\beta )+ (1-Y) \log (1-\boldsymbol{X} \cdot \boldsymbol\beta) \right)\\ &=& \frac{\partial}{\partial \beta_j} YX_i \frac{1}{(\boldsymbol{X} \cdot \boldsymbol\beta)} + (1-Y)X_i \frac{1}{ (1-\boldsymbol{X} \cdot \boldsymbol\beta)} \\ &=& - YX_i X_j\frac{1}{(\boldsymbol{X} \cdot \boldsymbol\beta)^2} - (1-Y)X_iX_j \frac{1}{ (1-\boldsymbol{X} \cdot \boldsymbol\beta)^2} \\ &=& -X_iX_j \left( \frac{Y}{(\boldsymbol{X} \cdot \boldsymbol\beta)^2} + \frac{1-Y}{(1-\boldsymbol{X} \cdot \boldsymbol\beta)^2} \right) \\ \end{array}$$

And for the Hessian of the likelihood of multiple observations you'd get

$$\begin{array}{rcl}\frac{\partial^2}{\partial \beta_i \partial \beta_j}\log \prod_{\forall k} {p}^{Y_k} (1-p_k)^{1-Y_k} &=& \sum_{\forall{k}} X_{ik}X_{jk} \left( \frac{Y_k}{(\boldsymbol{X_k} \cdot \boldsymbol\beta)^2} + \frac{1-Y_k}{(1-\boldsymbol{X_k} \cdot \boldsymbol\beta)^2} \right) \\ \end{array}$$

Your computation seems to go wrong where you are repeating the index $i$ in a term like $\frac{Y_i}{X_i^\intercal\beta}X_i$ but the $i$ relates to different indexes and your $X$ is actually a matrix with two indices, one indice for the observation number and one indice for the column/coefficient/parameter. The $X_i^T$ and $X_i$ are not just different in that the one is the transpose of the other, it should be $\frac{Y_i}{X_i^\intercal\beta}X_{ij}$ where $X$ is a matrix $X_i$ is a vector (relating to the $i$-th observation) and $X_{ij}$ is a scalar (relating to the ($j$-th component of $X$ for the $i$-th observation).

---

To compute the expectation... you have $Y=1$ and $H(l)= X_iX_j \frac{1}{(\boldsymbol{X} \cdot \boldsymbol\beta)^2}$ with probability $\boldsymbol{X} \cdot \boldsymbol\beta$ and you $Y=0$ and $H(l)= X_iX_j \frac{1}{(1-\boldsymbol{X} \cdot \boldsymbol\beta)^2}$ with probability $1-\boldsymbol{X} \cdot \boldsymbol\beta$. This makes

$$E(H(l)|\boldsymbol{X},\boldsymbol{\beta}) = -X_iX_j \left( \frac{1}{(\boldsymbol{X} \cdot \boldsymbol\beta)} + \frac{1}{(1-\boldsymbol{X} \cdot \boldsymbol\beta)} \right) = -\frac{X_iX_j }{(\boldsymbol{X} \cdot \boldsymbol\beta)(1-\boldsymbol{X} \cdot \boldsymbol\beta)} = -\frac{X_iX_j }{\text{Var}(Y|\boldsymbol{X},\boldsymbol{\beta})} $$