# Fisher information as negative log likelihood

The Fisher Information is defined as the covariance matrix, or $$E_{y \sim P(y;\theta)}[ \nabla_{\theta} ln(p(y;\theta)) \nabla_{\theta} ln(p(y;\theta))^T]$$.

It can also be defined as $$E_{y \sim P(y;\theta)}[ -\nabla^2_{\theta} ln p(y;\theta)]$$.

To show that both definitions are equal:

$$E_{y \sim P(y;\theta)}[ -\nabla^2_{\theta} ln ( p(y;\theta) )] = - \int^{\infty}_{-\infty} p(y) \nabla^2 ln ( p(y) ) dy \\$$ Taking the first derivative $$= - \int^{\infty}_{-\infty} p(y) \nabla(\frac{1}{p(y)} \nabla p(y)) dy \\$$ Taking the second derivative $$= - \int^{\infty}_{-\infty} p(y) (-\frac{1}{p(y)^2} \nabla p(y) \nabla p(y) + \frac{1}{p(y)} \nabla^2 p(y)) dy \\$$ Splitting into two integrals $$= \int^{\infty}_{-\infty} p(y) \frac{1}{p(y)^2} \nabla p(y) \nabla p(y) dy + \int^{\infty}_{-\infty} p(y) \frac{1}{p(y)} \nabla^2 p(y) dy \\$$ The first integral is equivalent to the covariance matrix, the second integral is equal to 0 $$= E_{y \sim P(y;\theta)}[ \nabla_{\theta} ln(p(y;\theta)) \nabla_{\theta} ln(p(y;\theta))] + 0$$

The question I have is, $$ln(p(y))$$ is a scalar. Taking its gradient w.r.t vector $$\theta \in R^{n \times 1}$$ results in a $$R^{n \times 1}$$ vector.

However, in the second last integral, we have $$\nabla p(y) \nabla p(y)$$.

If we take the pointwise product, the result is $$R^{n \times 1}$$.

If we take $$\nabla p(y)^T \nabla p(y)$$, the result is $$R^{1 \times 1}$$.

If we take the outer product, or $$\nabla p(y) \nabla p(y)^T$$, the result is the covariance matrix.

Why are we taking the outer product? How do we know we need the outer product?

We have a vector, $$\nabla p(y) = \mathbf u_{n\times 1}$$ multiplied with a scalar, $$v=1/p(y)$$, both parameterized by $$\theta$$, and we want to take its derivative wrt $$\theta$$, i.e.
$$\frac{\partial (v\mathbf u)}{\partial \theta}=v\frac{\partial \mathbf u}{\partial \theta} + \frac{\partial v}{\partial \theta}\mathbf u^T={1\over p(y)}\nabla^2p(y)-{1\over p(y)^2}\nabla p(y) \nabla p(y)^T$$