# $\frac\partial{\partial\beta}\left(\sum\frac{Y_i}{X_i^\intercal\beta}X_i-\sum\frac{1-Y_i}{1-X_i^\intercal\beta}X_i\right)$

How do you take the derivative of the function

$$s(\beta)=\displaystyle\sum\frac{Y_i}{X_i^\intercal\beta}X_i-\sum\frac{1-Y_i}{1-X_i^\intercal\beta}X_i?$$

Attempt:

$$H(\beta)=\frac\partial{\partial\beta}s(\beta)=\frac \partial {\partial \beta} \sum\left(\frac{Y_i}{X_i^\intercal\beta}X_i-\frac{1-Y_i}{1-X_i^\intercal\beta}X_i\right)=\\[2em]\sum\left(-\frac{Y_i}{(X_i^\intercal\beta)^2}X_iX_i^\intercal-\frac{1-Y_i}{(1-X_i^\intercal\beta)^2}X_iX_i^\intercal\right)$$

as well as calculate $$EH(\beta)$$?

I need this because I'm trying to calculate asymptotic variance of a maximum likelihood estimator $$-I(\beta)^{-1}=-E(H(\beta))^{-1}$$.

By the way, $$E(Y_i|X_i)=X_i^\intercal\beta$$ here if you want to take the solution all the way.

• Is $\beta$ a vector or a single coefficient? Based on the expression $X_i^T\beta$ (which is itselve also unclear), which looks like a vector product, I would suspect it is a vector. Could you clarify this. Commented Jan 21, 2021 at 6:21
• $X_i,\beta$ are vectors of length P
– user308286
Commented Jan 21, 2021 at 6:34
• while $Y_i\sim Bern(\pi_i)\in \{0,1\}$ and $\pi_i=X_i^\intercal\beta$, the identity link
– user308286
Commented Jan 21, 2021 at 6:49
• I took the expectation of the hessian and came up with an answer. It’s all good
– user308286
Commented Jan 21, 2021 at 7:13
• Is $s(\beta)$ a vector or a scalar? I imagine that $\frac{Y_i}{X_i^T\beta}$ is a scalar $\frac{Y_i}{\hat{Y}_i}$ (and by the way: $X_i^T\beta = \hat{Y}_i$ but not $X_i^T\beta = E(Y_i|X_i)$. Isn't it an estimate of the expectation and not equal to the expectation?). Then $\frac{Y_i}{X_i^T\beta} X_i$ is a vector. But... the Hessian is a derivative of a scalar valued function. Commented Jan 21, 2021 at 9:13

It seems like you are computing the Hessian of a logarithm of the likelihood $$\mathcal{L} = \prod p_i^{Y_i} (1-p_i)^{1-Y_i}$$ with $$p_i = X_i \cdot \beta$$.

For simplicity you might consider instead to compute the Hessian of a single observation $$i$$ because that subscript $$_i$$ seems to complicate the computation. $$\mathcal{l} = p^Y (1-p)^{1-Y}$$ with $$p = X \cdot \beta$$

$$\begin{array}{rcl}\frac{\partial^2}{\partial \beta_i \partial \beta_j} \log {p}^Y (1-p)^{1-Y} &=& \frac{\partial^2}{\partial \beta_i \partial \beta_j} \left( Y \log( \boldsymbol{X} \cdot \boldsymbol\beta )+ (1-Y) \log (1-\boldsymbol{X} \cdot \boldsymbol\beta) \right)\\ &=& \frac{\partial}{\partial \beta_j} YX_i \frac{1}{(\boldsymbol{X} \cdot \boldsymbol\beta)} + (1-Y)X_i \frac{1}{ (1-\boldsymbol{X} \cdot \boldsymbol\beta)} \\ &=& - YX_i X_j\frac{1}{(\boldsymbol{X} \cdot \boldsymbol\beta)^2} - (1-Y)X_iX_j \frac{1}{ (1-\boldsymbol{X} \cdot \boldsymbol\beta)^2} \\ &=& -X_iX_j \left( \frac{Y}{(\boldsymbol{X} \cdot \boldsymbol\beta)^2} + \frac{1-Y}{(1-\boldsymbol{X} \cdot \boldsymbol\beta)^2} \right) \\ \end{array}$$

And for the Hessian of the likelihood of multiple observations you'd get

$$\begin{array}{rcl}\frac{\partial^2}{\partial \beta_i \partial \beta_j}\log \prod_{\forall k} {p}^{Y_k} (1-p_k)^{1-Y_k} &=& \sum_{\forall{k}} X_{ik}X_{jk} \left( \frac{Y_k}{(\boldsymbol{X_k} \cdot \boldsymbol\beta)^2} + \frac{1-Y_k}{(1-\boldsymbol{X_k} \cdot \boldsymbol\beta)^2} \right) \\ \end{array}$$

Your computation seems to go wrong where you are repeating the index $$i$$ in a term like $$\frac{Y_i}{X_i^\intercal\beta}X_i$$ but the $$i$$ relates to different indexes and your $$X$$ is actually a matrix with two indices, one indice for the observation number and one indice for the column/coefficient/parameter. The $$X_i^T$$ and $$X_i$$ are not just different in that the one is the transpose of the other, it should be $$\frac{Y_i}{X_i^\intercal\beta}X_{ij}$$ where $$X$$ is a matrix $$X_i$$ is a vector (relating to the $$i$$-th observation) and $$X_{ij}$$ is a scalar (relating to the ($$j$$-th component of $$X$$ for the $$i$$-th observation).

To compute the expectation... you have $$Y=1$$ and $$H(l)= X_iX_j \frac{1}{(\boldsymbol{X} \cdot \boldsymbol\beta)^2}$$ with probability $$\boldsymbol{X} \cdot \boldsymbol\beta$$ and you $$Y=0$$ and $$H(l)= X_iX_j \frac{1}{(1-\boldsymbol{X} \cdot \boldsymbol\beta)^2}$$ with probability $$1-\boldsymbol{X} \cdot \boldsymbol\beta$$. This makes

$$E(H(l)|\boldsymbol{X},\boldsymbol{\beta}) = -X_iX_j \left( \frac{1}{(\boldsymbol{X} \cdot \boldsymbol\beta)} + \frac{1}{(1-\boldsymbol{X} \cdot \boldsymbol\beta)} \right) = -\frac{X_iX_j }{(\boldsymbol{X} \cdot \boldsymbol\beta)(1-\boldsymbol{X} \cdot \boldsymbol\beta)} = -\frac{X_iX_j }{\text{Var}(Y|\boldsymbol{X},\boldsymbol{\beta})}$$

• Thanks for putting together the answer! :-) (I think there’s one little error where you wrote $\prod\log$ for the hessian of multiple observations, should be $\log\prod$ or $\sum\log$.)
– user308286
Commented Jan 22, 2021 at 1:32