# Covariance matrix of parameters in logistic regression

Let $$f(y)=\prod_i \binom{n_i}{y_i}\pi(x_i)^{y_i}(1-\pi(x_i))^{n_i-y_i}$$ where $$\pi(x_i)=\frac{e^{\sum_j x_{ij}B_j}}{1+e^{\sum_j x_{ij}B_j}}$$ then the likelihood is $$L\propto \prod_{i}\pi(x_i)^{y_i}(1-\pi(x_i))^{(n_i-y_i)}$$ $$l=\sum_i y_i log(\frac{\pi(x_i)}{(1-\pi(x_i))})+\sum_i n_i log(1-\pi(x_i))$$ I found that $$\frac{d^2l}{dBdB}=-\sum_i n_ix_{ij}\frac{e^{\sum_j x_{ij}B_j}}{(1+e^{\sum_j x_{ij}B_j})^2}=-\sum_i n_ix_{ij}\pi(x_i)(1-\pi(x_i))$$ the Fisher information is $$I(\hat{B})=-E[\frac{d^2l}{dBdB}]$$ and $$cov(\hat{B})=I^{-1}(\hat{B})=(X'Diag[(n_i{\hat{\pi}}(x_i)(1-{\hat{\pi}}(x_i))]X)^{-1}(*)$$

In the last line $X$ is a matrix.

anyone can help me understand how to get $(*)$?

EDIT: $B$ is the parameters of the model

• What does $\hat{\pi}(x_i)$ denote in the last expression? And what is $B$? – Greenparker May 8 '16 at 22:08
• @Greenparker $\hat{\pi(x_i)}$ is the estimatives of $\pi(x_i)$ since you get the estimatives of the parameters $\hat{B}$ – user72621 May 8 '16 at 22:25

(Since this seems like a homework problem, I am just pointing out the essential details.)

You made a mistake in calculating derivates, and also ignored some important details. Your log-likelihood is correct.

$$l = \sum_i y_i \sum_j x_{ij} B_j - \sum_{i} n_i \log \left(1 + e^{\sum_j x_{ij}B_j} \right).$$

To calculate the $(l,k)$th entry of the covariance matrix we first take the derivate with respect to $l$ and then take a second derivative with respect to $k$.

$$\dfrac{\partial l}{\partial B_l} = \sum_{i} y_i x_{ij} - \sum_{i}n_i \dfrac{x_{il} e^{\sum_{j}x_{ij} B_j}}{1 + e^{\sum_{j}x_{ij} B_j}}.$$

$$\dfrac{\partial^2 l}{\partial B_l \partial B_k} = -\sum_{i} n_i x_{il}x_{ik}\dfrac{e^{\sum_{j}x_{ij} B_j}}{\left(1 + e^{\sum_{j}x_{ij} B_j} \right)^2} = -\sum_{i} n_i x_{il}x_{ik} \pi(x_i) (1- \pi(x_i)).$$

(You had missed the second $x_{ik}$ term. Thus the $(l,k)$th entry of the information matrix is, $$I_{l,k}(B) = \sum_{i} n_i x_{il}x_{ik} \pi(x_i) (1- \pi(x_i)).$$

Writing this in matrix form leads to the desired answer. Estimates for $\pi(x_i)$ can be substituted in the covariance matrix to obtain an estimate of the covariance.