The Fisher Information is defined as the covariance matrix, or $E_{y \sim P(y;\theta)}[ \nabla_{\theta} ln(p(y;\theta)) \nabla_{\theta} ln(p(y;\theta))^T]$.

It can also be defined as $E_{y \sim P(y;\theta)}[ -\nabla^2_{\theta} ln p(y;\theta)]$.

To show that both definitions are equal:

$$ E_{y \sim P(y;\theta)}[ -\nabla^2_{\theta} ln ( p(y;\theta) )] = - \int^{\infty}_{-\infty} p(y) \nabla^2 ln ( p(y) ) dy \\ $$ Taking the first derivative $$ = - \int^{\infty}_{-\infty} p(y) \nabla(\frac{1}{p(y)} \nabla p(y)) dy \\ $$ Taking the second derivative $$ = - \int^{\infty}_{-\infty} p(y) (-\frac{1}{p(y)^2} \nabla p(y) \nabla p(y) + \frac{1}{p(y)} \nabla^2 p(y)) dy \\ $$ Splitting into two integrals $$ = \int^{\infty}_{-\infty} p(y) \frac{1}{p(y)^2} \nabla p(y) \nabla p(y) dy + \int^{\infty}_{-\infty} p(y) \frac{1}{p(y)} \nabla^2 p(y) dy \\ $$ The first integral is equivalent to the covariance matrix, the second integral is equal to 0 $$ = E_{y \sim P(y;\theta)}[ \nabla_{\theta} ln(p(y;\theta)) \nabla_{\theta} ln(p(y;\theta))] + 0 $$

The question I have is, $ln(p(y))$ is a scalar. Taking its gradient w.r.t vector $\theta \in R^{n \times 1}$ results in a $R^{n \times 1}$ vector.

However, in the second last integral, we have $\nabla p(y) \nabla p(y)$.

If we take the pointwise product, the result is $R^{n \times 1}$.

If we take $\nabla p(y)^T \nabla p(y)$, the result is $R^{1 \times 1}$.

If we take the outer product, or $\nabla p(y) \nabla p(y)^T$, the result is the covariance matrix.

Why are we taking the outer product? How do we know we need the outer product?


1 Answer 1


It's because of layout confusion. You're using denominator layout, and should be expanding the product according to it:

We have a vector, $\nabla p(y) = \mathbf u_{n\times 1}$ multiplied with a scalar, $v=1/p(y)$, both parameterized by $\theta$, and we want to take its derivative wrt $\theta$, i.e.

$$\frac{\partial (v\mathbf u)}{\partial \theta}=v\frac{\partial \mathbf u}{\partial \theta} + \frac{\partial v}{\partial \theta}\mathbf u^T={1\over p(y)}\nabla^2p(y)-{1\over p(y)^2}\nabla p(y) \nabla p(y)^T$$

The above equation is taken from wikipedia.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.