3
$\begingroup$

I'm working at Score test realization and I need to calculate the Fisher information in basic logistic model, $$\text{Logit}(\Pr(Y_i=1))=\beta_0+\beta_1X_i$$

And I have stuck at the calculation of this expectation: $$I=E\big(\sum_iX_i^2f(\beta_0+\beta_1X_i)(1-f(\beta_0+\beta_1X_i)\big)$$ such that $ f=\frac1{1+e^{-\beta_0-\beta_1X_i} }$.

Maybe someone has already faced this same problem?

Calculations:

Hypothesis: $$H_0: \beta_s=0 \text{vs. } H_1: \beta_s\neq0$$ Likelihood function: $l=\Pi_{i=1}^{n} (f_i)$

Statistics: \begin{align} U(\beta_s)&=\frac{\partial\big(\ln L(\beta_s)\big)}{\partial\beta_s}\\ V(\beta_s)&=-E\bigg( \frac{\partial^2\big(\ln L(\beta_s)\big)}{\partial\beta_s^2}\bigg) \end{align}

Calculating U statistic: \begin{align} L(Y_i,\beta_s)&=\ln(l(Y_i,\beta_s))\\ &=\ln\big( \Pi_i f_i(Y_i,\beta_s)\big)\\ &=\sum_i \ln f_i(Y)={\sum_i \bigg( \ln\left(Pr(Y=1)\big)^{Y_i} \big(Pr(Y=0)\right)^{(1-Y_i)} \bigg)} \\ &=\sum_i \left( Y_i\ln\big(Pr(Y_i=1)\big) + (1-Y_i)\ln\left(1-Pr(Y_i=1)\right) \right) \\ &=\sum_i \left( Y_i\ln\left(f(\beta_0+\beta_sX_{s,i})\right) + (1-Y_i)\ln\left(1-f(\beta_0+\beta_sX_{s,i})\right) \right)\\ &= \sum_i \bigg((\beta_0+\beta_sX_{s,i})(Y_i-1)+\ln(\frac{1}{1+e^{-(\beta_0+\beta_sX_{s,i})}} \bigg)\\ &= \sum_i \bigg((\beta_0+\beta_sX_{s,i})(Y_i-1)-\ln(1+e^{-(\beta_0-\beta_sX_{s,i})} \bigg) \end{align}

Taking the derivative by $\beta_s$: ${ \sum_i \bigg(X_{s,i}\big(Y_i - f(\beta_0+\beta_sX_{s,i})\big) \bigg)}$

At least: $V$ statistic is the expectation of the derivative of $U$ by $\beta_s$, \begin{align} U(\beta_s)'_{\beta_s}&=\bigg( \sum_i X_{s,i}\big(Y_i - f(\beta_0+\beta_sX_{s,i})\big) \bigg)'_{\beta_s}\\ &=\sum_i \bigg( X_{s,i}\big(Y_i - \frac{1}{1+e^{-(\beta_0-\beta_sX_{s,i})}}\big)'_{\beta_s} \bigg)\\ &=\sum_i \bigg( X_{s,i}(-1)(-1)\frac{-X_{s,i}e^{-\beta_0-\beta_sX_{s,i}}}{\big( 1+e^{-(\beta_0-\beta_sX_{s,i})}\big) ^2} \bigg)\\ &=\sum_i -X_{s,i}^2\bigg(\frac{e^{-\beta_0-\beta_sX_{s,i}}}{\big( 1+e^{-(\beta_0-\beta_sX_{s,i})}\big) ^2} \bigg) \end{align}

For every $\xi$, $$\frac{\xi}{(1+\xi)^2}=\frac{1}{1+\xi}\big( 1- \frac{1}{1+\xi} \big)$$

and thus \begin{align} U(\beta_s)'_{\beta_s}&=-\sum_i X_{s,i}^2\frac{1}{1+e^{-(\beta_0-\beta_sX_{s,i})}}\left( 1-\frac{1}{1+e^{-(\beta_0-\beta_sX_{s,i})}} \right) \\ &=-\sum_i X_{s,i}^2 f(\beta_0+\beta_sX_{s,i}) \big( 1-f(\beta_0+\beta_sX_{s,i}) \big) \end{align}

Now I should take expectation of this and I don't know how to do it.

Improve this question
$\endgroup$
7
  • $\begingroup$ Shouldn't you be using the log likelihood? $\endgroup$
    – VCG
    Aug 23, 2016 at 18:14
  • $\begingroup$ If the information is as you claim, then you can eliminate the expectation, because (in a traditional regression setting) it is taken over $Y|X$, and $Y$ is not present. But, I echo @frage_man's concern about earlier errors; could you add a summary of your work to this point? $\endgroup$
    – eric_kernfeld
    Aug 23, 2016 at 18:27
  • $\begingroup$ I added calculations. $\endgroup$
    – Kess
    Aug 23, 2016 at 20:18
  • $\begingroup$ So I think your derivations are correct. So the Fisher info is evaluated at the MLE from the FOC, which here doesn't have a closed form solution. So I'm not sure what you expect to get. $\endgroup$
    – VCG
    Aug 23, 2016 at 22:10
  • $\begingroup$ Thank you. Sorry, I don't know what FOC is. I found the realization, where Fisher info is evaluated as $\sum_i \bigg((X_{s,i}^2- \bar X)\bar Y \big( 1-\bar Y) \big) \bigg)$ (Where $ \bar Y$ is an estimate for $f(\beta_0+X \beta_1)$). I wanted to receive the same result and current result is close to expected. $\endgroup$
    – Kess
    Aug 23, 2016 at 22:54

1 Answer 1

Reset to default
2
$\begingroup$

All calculations was correct. I forgot that Fisher info formula is $-E\bigg( \frac{\partial^2\big(lnL(\beta_s)\big)}{\partial\beta_s^2}\bigg)$ only in regular models. So to get the right answer we must center $X$, and then, as @eric_kernfeld told, eliminate expectation.

The correct result is $\sum_i \bigg((X_{s,i}-\bar X)^2 f(\beta_0+\beta_sX_{s,i}) \big( 1-f(\beta_0+\beta_sX_{s,i}) \big) \bigg)$

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.