Fisher information as negative log likelihood

Question

The Fisher Information is defined as the covariance matrix, or $E_{y \sim P(y;\theta)}[ \nabla_{\theta} ln(p(y;\theta)) \nabla_{\theta} ln(p(y;\theta))^T]$.

It can also be defined as $E_{y \sim P(y;\theta)}[ -\nabla^2_{\theta} ln p(y;\theta)]$.

To show that both definitions are equal:

$$ E_{y \sim P(y;\theta)}[ -\nabla^2_{\theta} ln ( p(y;\theta) )] = - \int^{\infty}_{-\infty} p(y) \nabla^2 ln ( p(y) ) dy \\ $$ Taking the first derivative $$ = - \int^{\infty}_{-\infty} p(y) \nabla(\frac{1}{p(y)} \nabla p(y)) dy \\ $$ Taking the second derivative $$ = - \int^{\infty}_{-\infty} p(y) (-\frac{1}{p(y)^2} \nabla p(y) \nabla p(y) + \frac{1}{p(y)} \nabla^2 p(y)) dy \\ $$ Splitting into two integrals $$ = \int^{\infty}_{-\infty} p(y) \frac{1}{p(y)^2} \nabla p(y) \nabla p(y) dy + \int^{\infty}_{-\infty} p(y) \frac{1}{p(y)} \nabla^2 p(y) dy \\ $$ The first integral is equivalent to the covariance matrix, the second integral is equal to 0 $$ = E_{y \sim P(y;\theta)}[ \nabla_{\theta} ln(p(y;\theta)) \nabla_{\theta} ln(p(y;\theta))] + 0 $$

---

The question I have is, $ln(p(y))$ is a scalar. Taking its gradient w.r.t vector $\theta \in R^{n \times 1}$ results in a $R^{n \times 1}$ vector.

However, in the second last integral, we have $\nabla p(y) \nabla p(y)$.

If we take the pointwise product, the result is $R^{n \times 1}$.

If we take $\nabla p(y)^T \nabla p(y)$, the result is $R^{1 \times 1}$.

If we take the outer product, or $\nabla p(y) \nabla p(y)^T$, the result is the covariance matrix.

Why are we taking the outer product? How do we know we need the outer product?

Answer 1

It's because of layout confusion. You're using denominator layout, and should be expanding the product according to it:

We have a vector, $\nabla p(y) = \mathbf u_{n\times 1}$ multiplied with a scalar, $v=1/p(y)$, both parameterized by $\theta$, and we want to take its derivative wrt $\theta$, i.e.

$$\frac{\partial (v\mathbf u)}{\partial \theta}=v\frac{\partial \mathbf u}{\partial \theta} + \frac{\partial v}{\partial \theta}\mathbf u^T={1\over p(y)}\nabla^2p(y)-{1\over p(y)^2}\nabla p(y) \nabla p(y)^T$$

The above equation is taken from wikipedia.