# Fisher information in logit model

I'm working at Score test realization and I need to calculate the Fisher information in basic logistic model, $$\text{Logit}(\Pr(Y_i=1))=\beta_0+\beta_1X_i$$

And I have stuck at the calculation of this expectation: $$I=E\big(\sum_iX_i^2f(\beta_0+\beta_1X_i)(1-f(\beta_0+\beta_1X_i)\big)$$ such that $$f=\frac1{1+e^{-\beta_0-\beta_1X_i} }$$.

Maybe someone has already faced this same problem?

Calculations:

Hypothesis: $$H_0: \beta_s=0 \text{vs. } H_1: \beta_s\neq0$$ Likelihood function: $$l=\Pi_{i=1}^{n} (f_i)$$

Statistics: \begin{align} U(\beta_s)&=\frac{\partial\big(\ln L(\beta_s)\big)}{\partial\beta_s}\\ V(\beta_s)&=-E\bigg( \frac{\partial^2\big(\ln L(\beta_s)\big)}{\partial\beta_s^2}\bigg) \end{align}

Calculating U statistic: \begin{align} L(Y_i,\beta_s)&=\ln(l(Y_i,\beta_s))\\ &=\ln\big( \Pi_i f_i(Y_i,\beta_s)\big)\\ &=\sum_i \ln f_i(Y)={\sum_i \bigg( \ln\left(Pr(Y=1)\big)^{Y_i} \big(Pr(Y=0)\right)^{(1-Y_i)} \bigg)} \\ &=\sum_i \left( Y_i\ln\big(Pr(Y_i=1)\big) + (1-Y_i)\ln\left(1-Pr(Y_i=1)\right) \right) \\ &=\sum_i \left( Y_i\ln\left(f(\beta_0+\beta_sX_{s,i})\right) + (1-Y_i)\ln\left(1-f(\beta_0+\beta_sX_{s,i})\right) \right)\\ &= \sum_i \bigg((\beta_0+\beta_sX_{s,i})(Y_i-1)+\ln(\frac{1}{1+e^{-(\beta_0+\beta_sX_{s,i})}} \bigg)\\ &= \sum_i \bigg((\beta_0+\beta_sX_{s,i})(Y_i-1)-\ln(1+e^{-(\beta_0-\beta_sX_{s,i})} \bigg) \end{align}

Taking the derivative by $$\beta_s$$: $${ \sum_i \bigg(X_{s,i}\big(Y_i - f(\beta_0+\beta_sX_{s,i})\big) \bigg)}$$

At least: $$V$$ statistic is the expectation of the derivative of $$U$$ by $$\beta_s$$, \begin{align} U(\beta_s)'_{\beta_s}&=\bigg( \sum_i X_{s,i}\big(Y_i - f(\beta_0+\beta_sX_{s,i})\big) \bigg)'_{\beta_s}\\ &=\sum_i \bigg( X_{s,i}\big(Y_i - \frac{1}{1+e^{-(\beta_0-\beta_sX_{s,i})}}\big)'_{\beta_s} \bigg)\\ &=\sum_i \bigg( X_{s,i}(-1)(-1)\frac{-X_{s,i}e^{-\beta_0-\beta_sX_{s,i}}}{\big( 1+e^{-(\beta_0-\beta_sX_{s,i})}\big) ^2} \bigg)\\ &=\sum_i -X_{s,i}^2\bigg(\frac{e^{-\beta_0-\beta_sX_{s,i}}}{\big( 1+e^{-(\beta_0-\beta_sX_{s,i})}\big) ^2} \bigg) \end{align}

For every $$\xi$$, $$\frac{\xi}{(1+\xi)^2}=\frac{1}{1+\xi}\big( 1- \frac{1}{1+\xi} \big)$$

and thus \begin{align} U(\beta_s)'_{\beta_s}&=-\sum_i X_{s,i}^2\frac{1}{1+e^{-(\beta_0-\beta_sX_{s,i})}}\left( 1-\frac{1}{1+e^{-(\beta_0-\beta_sX_{s,i})}} \right) \\ &=-\sum_i X_{s,i}^2 f(\beta_0+\beta_sX_{s,i}) \big( 1-f(\beta_0+\beta_sX_{s,i}) \big) \end{align}

Now I should take expectation of this and I don't know how to do it.

• Shouldn't you be using the log likelihood?
– VCG
Commented Aug 23, 2016 at 18:14
• If the information is as you claim, then you can eliminate the expectation, because (in a traditional regression setting) it is taken over $Y|X$, and $Y$ is not present. But, I echo @frage_man's concern about earlier errors; could you add a summary of your work to this point? Commented Aug 23, 2016 at 18:27
– Kess
Commented Aug 23, 2016 at 20:18
• So I think your derivations are correct. So the Fisher info is evaluated at the MLE from the FOC, which here doesn't have a closed form solution. So I'm not sure what you expect to get.
– VCG
Commented Aug 23, 2016 at 22:10
• Thank you. Sorry, I don't know what FOC is. I found the realization, where Fisher info is evaluated as $\sum_i \bigg((X_{s,i}^2- \bar X)\bar Y \big( 1-\bar Y) \big) \bigg)$ (Where $\bar Y$ is an estimate for $f(\beta_0+X \beta_1)$). I wanted to receive the same result and current result is close to expected.
– Kess
Commented Aug 23, 2016 at 22:54

All calculations was correct. I forgot that Fisher info formula is $-E\bigg( \frac{\partial^2\big(lnL(\beta_s)\big)}{\partial\beta_s^2}\bigg)$ only in regular models. So to get the right answer we must center $X$, and then, as @eric_kernfeld told, eliminate expectation.
The correct result is $\sum_i \bigg((X_{s,i}-\bar X)^2 f(\beta_0+\beta_sX_{s,i}) \big( 1-f(\beta_0+\beta_sX_{s,i}) \big) \bigg)$